Savepoints

Engine automatically creates savepoints of your running executions. Savepoints are used to scale your automations and can be used to recover an execution after an error.

Use Cases

Savepoints are used to:

• Persist the state of executions when they are idle.
• Protect executions from workspace process downtimes.
• Transfer executions between workspace processes to allow for scaling.
• Mark a position in your code which can be used to jump back to.

note

Engine automatically handles the persisting, scaling and restoring for you.

Concept

Savepoints contain all information needed for the execution to resume at exactly the same instruction and going forward. Additionally the line and name of the Flow-API call and the timestamp is stored.

It is possible to revert an execution to the state of a savepoint for it to resume operation from the instruction. The complete state of the execution is restored, including all variables used in the execution.

warning

Restoring an execution to a savepoint does not however affect other objects, like Engine settings, Engine files, or third party systems. Only the execution is restored. Objects which the execution already modified will remain modified.

Types of Savepoints

Additionally to the automatically created implicit savepoints, a Flow script can explicitly request a savepoint to be created. Such explicit savepoints can also be assigned a name. See Create a Named Savepoint.

Using Savepoints

Persisting Idle Executions

When an execution is idle for more than 60 seconds Engine will store its state in the database and unload it from memory. See SLEEP_MAX_SECONDS in Performance Settings

When the execution resumes, Engine will automatically restore the state.

note

Since savepoints are created implicitly with each Flow-API call, executions are resistant to workspace process crashes, server reboots, software upgrades, or network interruptions. As soon as a workspace process comes online again, it will resume the execution from the last state.

Transfer Executions for Scaling

Engine workspace processes are stateless workers. They consume executions from the database, process them, and and save the state in the database. Multiple workspace processes can be active at the same time. Whenever possible the least-busy workspace process is chosen to process an execution.

note

When deploying Engine in an auto-scaling environment like Kubernetes, the CPU usage of workspace processes can be used to determine the need for up- or downscaling.

Create a Named Savepoint

The Flow-API command create_savepoint creates an explicit, named savepoint.

example

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # fetch some data
    data = this.connect(
        'database',
        fetch='SELECT * FROM data',
    ).get('output_value')['result']

    # create a named savepoint
    # in case of an error the execution can be reverted
    # to this state and resume normal operations
    this.create_savepoint('with-fetched-data')

    # process the data
    # in case the remote-api call fails it is possible to
    # revert to the savepoint above and retry the remote-api call
    # without having to fetch the data again
    this.connect(
        'remote-api',
        method='post',
        path='/upload',
        json=data,
    )
    return this.success('all done')

Restore an Execution to a Savepoint

It is possible to restore a savepoint using the Flow-API:

this.load_savepoint('with-fetched-data', by='name')
# or by ID
this.load_savepoint('8f47bbb1-6a04-435f-952f-1cbb84f71b4a')

It is also possible to load a savepoints with a button click via the savepoints tab in the UI. Every Execution has a tab called Savepoints where a list with Created at, Line number, Action and Command shows the available savepoints. They can be loaded easily with a click.

warning

Be careful to avoid loops when restoring a Savepoint using load_savepoint.

note

When restoring an execution to a previous state, any timeout which the execution currently has set will be cleared.

Savepoint lifetime

You can configure the workspace-wide savepoint retention of implicit savepoints in the workspace settings. Either all implicit savepoints of an execution are preserved or just the last.

You can overwrite the workspace-wide savepoint retention of individual flows, by navigating to the flow and selecting "Change savepoint retention" from the options menu.

the button to change savepoint retention of individual flows

note

Explicit (manual) savepoints are always retained.

warning

When an execution is deleted all its savepoints are deleted as well.

Learn More

Exceptions

Workspace configuration