Cloudomation automatically creates savepoints of your running executions. Savepoints are used to scale your automations and can be used to recover an execution after an error.
Savepoints are used to:
- Persist the state of executions when they are idle.
- Protect executions from workspace process downtimes.
- Transfer executions between workspace processes to allow for scaling.
- Mark a position in your code which can be used to jump back to.
Cloudomation automatically handles the persisting, scaling and restoring for you.
Savepoints contain all information needed for the execution to resume at exactly the same instruction and going forward. Additionally the line and name of the Flow-API call and the timestamp is stored.
It is possible to revert an execution to the state of a savepoint for it to resume operation from the instruction. The complete state of the execution is restored, including all variables used in the execution.
Restoring an execution to a savepoint does not however affect other objects, like Cloudomation settings, Cloudomation files, or third party systems. Only the execution is restored. Objects which the execution already modified will remain modified.
Types of Savepoints
Additionally to the automatically created implicit savepoints, a Flow script can explicitly request a savepoint to be created. Such explicit savepoints can also be assigned a name. See Create a Named Savepoint.
Persisting Idle Executions
When an execution is idle for more than 60 seconds Cloudomation will store its state in the database and unload it from memory. See
SLEEP_MAX_SECONDS in Performance Settings
When the execution resumes, Cloudomation will automatically restore the state.
Since savepoints are created implicitly with each Flow-API call, executions are resistant to workspace process crashes, server reboots, software upgrades, or network interruptions. As soon as a workspace process comes online again, it will resume the execution from the last state.
Transfer Executions for Scaling
Cloudomation workspace processes are stateless workers. They consume executions from the database, process them, and and save the state in the database. Multiple workspace processes can be active at the same time. Whenever possible the least-busy workspace process is chosen to process an execution.
When deploying Cloudomation in an auto-scaling environment like Kubernetes, the CPU usage of workspace processes can be used to determine the need for up- or downscaling.
Create a Named Savepoint
The Flow-API command
create_savepoint creates an explicit, named savepoint.
def handler(system: flow_api.System, this: flow_api.Execution):
# fetch some data
data = this.connect(
fetch='SELECT * FROM data',
# create a named savepoint
# in case of an error the execution can be reverted
# to this state and resume normal operations
# process the data
# in case the remote-api call fails it is possible to
# revert to the savepoint above and retry the remote-api call
# without having to fetch the data again
return this.success('all done')
Restore an Execution to a Savepoint
It is possible to restore a savepoint using the Flow-API:
# or by ID
It is also possible to load a savepoints with a button click via the savepoints
tab in the UI. Every Execution has a tab called Savepoints where a list with
Last Command shows the available savepoints.
They can be loaded easily with a click.
Be careful to avoid loops when restoring a Savepoint using
When restoring an execution to a previous state, any timeout which the execution currently has set will be cleared.
When an execution is deleted all its savepoints are deleted as well.