Connection Resilience

About

Connections run outside Cloudomation Engine — on a remote host, a database, an HTTP service, and so on. The link between Cloudomation and that system can be lost while a connection is running: a network glitch, or a workspace restart (for example when Cloudomation itself is updated).

Cloudomation Engine is designed to survive these interruptions. When the link is lost and later re-established, each connection does one of two things depending on what it is doing:

• Reattach — the remote work keeps running and Cloudomation reconnects to it, then reports its real result. Nothing is lost and nothing runs twice.
• Re-run — the connection is started again from the beginning. This is only safe when repeating the operation has no duplicate side effects (it is idempotent).
• Cancel — the connection ends and the remote work is stopped.

note

This is about losing the connection to Cloudomation (network glitch, workspace restart). A user cancel of an execution is different: it always terminates the remote work, regardless of the settings described here.

Connectors that reattach

Some connectors start their work in a way that survives a lost connection and can be reattached later:

• SSH in execute_script mode — the script runs as a detached process on the host, tracked by a pidfile and an exit-code file. On reconnect Cloudomation reattaches to the still-running script (or reads its recorded exit code if it already finished).
• PS (WinRM) in run_powershell_script / run_batch_script mode — the script runs as a scheduled task on the Windows host. On reconnect Cloudomation reattaches to the running task.

These connectors reattach automatically — you do not need to configure anything.

The idempotent input

Connectors that cannot reattach (a REST call, a SQL query, SSH run_commands, PS execute_command, ...) decide between re-run and cancel based on whether the operation is idempotent — i.e. whether running it again from the start is safe.

Every connector accepts an optional idempotent input:

• idempotent=True — re-run from the start on reconnect.
• idempotent=False — cancel on reconnect.
• unset (default) — let the connector decide based on the operation.

this.connect(
    connector_type='REST',
    **flow_api.split_url('https://api.example.com/reports/build'),
    mode={'mode_name': 'post'},
    # a POST is not idempotent by default; force re-run only if you know it is safe
    idempotent=True,
)

Connector defaults when idempotent is unset

Connector	Default idempotency
REST	By HTTP method — GET, HEAD, OPTIONS, PUT, DELETE are idempotent; POST, PATCH are not.
FTP / SFTP / SMB / WebDAV	Read-only operations (list, download, read) are idempotent; operations that upload, delete, move or rename are not.
LDAP	search is idempotent; operations that add, modify or delete are not.
SQL (PostgreSQL, MySQL, MSSQL, Oracle)	Not idempotent. The mode does not tell whether a statement mutates — a fetch can run UPDATE ... RETURNING. Set idempotent=True explicitly for read-only queries.
Others	Not idempotent by default. Set idempotent=True when you know the operation is safe to repeat.

tip

When in doubt, leave idempotent unset. The conservative default is to cancel rather than risk running a mutating operation twice.

Workspace restarts

When a workspace process shuts down (for example during an update), running connections are not force-cancelled. Reattach-capable and idempotent connections are handed off so that another workspace process — or the restarted process — picks them up and reconnects or re-runs. This is what allows a Cloudomation workspace to update itself: the deployment runs over SSH execute_script, detaches when the workspace restarts, and reattaches afterwards to report the real result.