Connection Resilience

About

Connections run outside Cloudomation Engine — on a remote host, a database, an HTTP service, and so on. The link between Cloudomation and that system can be lost while a connection is running: a network glitch, or a workspace restart (for example when Cloudomation itself is updated).

Cloudomation Engine is designed to survive these interruptions. When the link is lost and later re-established, each connection does one of two things depending on what it is doing:

• Reattach — the remote work keeps running and Cloudomation reconnects to it, then reports its real result. Nothing is lost and nothing runs twice.
• Re-run — the connection is started again from the beginning. This is only safe when repeating the operation has no duplicate side effects (it is idempotent).
• Cancel — the connection ends and the remote work is stopped.

note

This is about losing the connection to Cloudomation (network glitch, workspace restart). A user cancel of an execution is different: it always terminates the remote work, regardless of the settings described here.

Connectors that reattach

Some connectors start their work in a way that survives a lost connection and can be reattached later:

• SSH in execute_script mode — the script runs as a detached process on the host, tracked by a pidfile and an exit-code file. On reconnect Cloudomation reattaches to the still-running script (or reads its recorded exit code if it already finished).
• PS (WinRM) in run_powershell_script / run_batch_script mode — the script runs as a scheduled task on the Windows host. On reconnect Cloudomation reattaches to the running task.

These connectors reattach automatically — you do not need to configure anything.

The idempotent input

Connectors that cannot reattach (a REST call, a SQL query, SSH run_commands, PS execute_command, ...) decide between re-run and cancel based on whether the operation is idempotent — i.e. whether running it again from the start is safe.

Every connector accepts an optional idempotent input:

• idempotent=True — re-run from the start on reconnect.
• idempotent=False — cancel on reconnect.
• unset (default) — let the connector decide based on the operation.

this.connect(
    connector_type='REST',
    **flow_api.split_url('https://api.example.com/reports/build'),
    mode={'mode_name': 'post'},
    # a POST is not idempotent by default; force re-run only if you know it is safe
    idempotent=True,
)

Connector defaults when idempotent is unset

Connector	Default idempotency
REST	By HTTP method — GET, HEAD, OPTIONS, PUT, DELETE are idempotent; POST, PATCH are not.
FTP / SFTP / SMB / WebDAV	Read-only operations (list, download, read) are idempotent; operations that upload, delete, move or rename are not.
LDAP	search is idempotent; operations that add, modify or delete are not.
SQL (PostgreSQL, MySQL, MSSQL, Oracle)	Not idempotent. The mode does not tell whether a statement mutates — a fetch can run UPDATE ... RETURNING. Set idempotent=True explicitly for read-only queries.
Others	Not idempotent by default. Set idempotent=True when you know the operation is safe to repeat.

tip

When in doubt, leave idempotent unset. The conservative default is to cancel rather than risk running a mutating operation twice.

Workspace restarts

When a workspace process shuts down (for example during an update), running connections are not force-cancelled. Reattach-capable and idempotent connections are handed off so that another workspace process — or the restarted process — picks them up and reconnects or re-runs. This is what allows a Cloudomation workspace to update itself: the deployment runs over SSH execute_script, detaches when the workspace restarts, and reattaches afterwards to report the real result.

Abrupt shutdowns

A graceful shutdown hands running connections off cleanly, as described above. An abrupt shutdown — a process crash, or an out-of-memory kill — stops a workspace process without that handoff. Cloudomation records when a connection has been dispatched to its remote system, so when the interrupted execution is picked up again it knows the connection had already started once.

What happens next depends on whether the connection is safe to resume:

• Reattach-capable connections (SSH execute_script, PS run_powershell_script / run_batch_script) and idempotent connections are resumed normally — reattached, or re-run from the start.
• A non-idempotent connection that had already started is not re-run. Re-running it could duplicate a side effect the first run already applied (a second database INSERT, a second POST), so the execution ends in error with a message explaining that it was interrupted and cannot be safely resumed. This gives non-idempotent connections at-most-once execution even across an abrupt kill.

If you know the operation is safe to repeat, set idempotent=True on the connection to allow the automatic re-run instead of the error.

note

The interrupted execution ends in error rather than silently repeating the operation. Re-run it yourself once you have confirmed the state of the remote system — or set idempotent=True if repeating it is always safe.