Settings and Locks
With settings it is possible to store structured data in Engine. Locks can be used to synchronize the processing of parallel running executions.
Use Cases
Settings can be used
- to store configuration parameters which are used by multiple flows
- by executions to read input, store outputs/logging/reports
- as lock objects
Concept
Each setting has a name and can contain any JSON-serializable data structure. Comment lines (lines starting with a hash sign) will be discarded and the order of key-value pairs will be sorted alphabetically upon saving. Settings are stored as JSON data structures and therefore behave differently than strings.
Settings can be accessed by ID or name. The value of a setting can only be read or written as a whole.
Settings also double as lock objects.
Using Settings
Settings can be manipulated via the user interface, via the REST API, and via flow scripts. The examples in this document are limited to one method per use case. The method described is interchangeable with any of the other methods.
To manipulate Settings using the command line you need an authorization token. Please see the Authentication documentation on how to obtain an authorization token.
Since settings are stored as JSON data structures, you can't just save any value. For example lines starting with the # symbol get discarded upon saving.
Store Configuration Parameters
You can manually or automatically store configuration parameters in settings which can be read by executions.
Store configuration parameters using the command line:
$ curl -X POST 'https://<my-workspace-name>.cloudomation.com/api/latest/setting' -d '{"name":"notification_emails","value":["toni@example.com","cory@example.com"]}' -H "Authorization: $TOKEN"
Read the configuration parameter in a flow:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# we use the setting named "notification_emails" and access its value
emails = system.setting('notification_emails').get('value')
this.connect(
'my-smtp-server',
name='send notification',
mode={
'mode_name': 'send_email',
'from_': 'noreply@example.com',
'to': emails,
'subject': 'notification from Engine',
'text': 'test email content',
},
)
return this.success('all done')
If you choose to change the emails which should receive notifications you only need to update it in one place: the setting value:
$ curl -X PATCH 'https://<my-workspace-name>.cloudomation.com/api/latest/setting/notification_emails?by=name' -d '{"value":["toni@example.com","cory@example.com","tracy@example.com"]}' -H "Authorization: $TOKEN"
and with the next execution your flow scripts will read and use the new value.
Store Outputs/Logging/Reports
Your flow scripts can write the value of a setting to store the result of some processing, store logging of some processing, or store a report which was generated:
Storing the result of some processing in a setting:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# do some processing
result = 42
# store the result
system.setting('occurrences_found').save(value=result)
return this.success('all done')
The save method overwrites existing records with the same name and type, unless they're read-only. If that is not intended, you can first check for the existence
of a record with the same name and type, before calling the save method.
Other flow scripts can read the value and adapt their behaviour accordingly:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
count = system.setting('occurrences_found').get('value')
if count > 32:
this.connect(
connector_type='SMTP',
host='mail.example.com',
mode={
'mode_name': 'send_email',
'from_': 'no-reply@example.com',
'to': ['kevin@example.com'],
'subject': 'counter alert',
'text': f'found {count} occurrences',
},
)
return this.success('all done')
The value can also be retrieved using the REST API:
$ curl 'https://<my-workspace-name>.cloudomation.com/api/latest/setting/occurrences_found?by=name' -H "Authorization: $TOKEN" | jq .
{
"setting": {
"name": "occurences_found",
"value": 42,
...
}
}
:::
Lock Objects
It is possible for an execution to acquire a lock on a setting.
Each setting can be locked by one execution at a time. Other executions waiting to acquire a lock on the same setting will wait in the status WAITING_LOCK until it becomes available or a timeout occurs.
Only exclusive locks on settings can be acquired. There is no shared-lock mechanism.
Make sure only one cloud-vm is provisioned at once.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# we try to acquire the lock
system.setting('cloud-vm-lock').acquire()
# start the flow which launches the cloud-vm
system.flow('create-cloud-vm').run()
# use the cloud-vm
this.connect(
'cloud-vm',
mode={'mode_name': 'execute_script', 'script': 'sleep 30'},
)
# delete the cloud-vm
system.flow('remove-cloud-vm').run()
# free the lock
system.setting('cloud-vm-lock').release()
return this.success('all done')
If the cloud-vm-lock is free when an execution of this flow runs, the execution will acquire the lock and continue its processing.
When a second execution of the flow is started during the processing, it cannot acquire the lock and waits in the status WAITING_LOCK. A lock wait timeout of 60 seconds is used by default. If the lock cannot be acquired in this time, the second execution fails with a LockTimeoutError exception. Pass a different wait_timeout (in seconds) to acquire() or lock() to change this limit, or wait_timeout=None to wait without a timeout.
If instead the lock becomes free within the timeout, the second execution immediately acquires it and continues its processing.
If several executions are waiting for the same lock, the order in which they acquire the lock is not defined.
Releasing a lock automatically
lock() returns a context manager that acquires the lock when the with block is entered and releases it when the block exits, including when the block raises an exception. This is the recommended way to hold a lock, because the lock is released even if the code between acquiring and releasing fails:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
with system.setting('cloud-vm-lock').lock() as setting:
system.flow('create-cloud-vm').run()
this.connect(
'cloud-vm',
mode={'mode_name': 'execute_script', 'script': 'sleep 30'},
)
system.flow('remove-cloud-vm').run()
# the lock is released here, even if the block above raised
return this.success('all done')
lock() accepts the same wait_timeout argument as acquire().
Checking whether a setting is locked
is_locked() returns True when the setting is held by an execution and False otherwise:
if system.setting('cloud-vm-lock').is_locked():
return this.success('a cloud-vm is already being provisioned')
The result reflects the setting data loaded on the object at the time it was fetched; re-read the setting to check the current state.
Re-entrancy and automatic release
An execution that already holds a lock can acquire the same lock again without waiting. The lock is tracked per execution, and a single release() frees it.
A lock is also released automatically when the execution holding it ends, in any end status. An execution that fails or is cancelled while holding a lock does not leave the setting locked — the executions waiting for it are resumed once the holder ends.
Deadlocks
A deadlock occurs when executions form a cycle of lock waits: each execution in the cycle holds a lock that the next one waits for, so none of them can proceed. Instead of letting every execution in the cycle wait until its timeout elapses, the platform detects the cycle as soon as it forms and fails the execution whose acquire attempt closed the cycle with a DeadlockError.
Because it is raised the moment the cyclic wait is detected rather than after a timeout, DeadlockError is distinct from LockTimeoutError. Acquiring locks in a consistent order across flows avoids this class of deadlock.
Deadlock detection covers cycles formed purely of setting-lock waits. A cycle routed through a different kind of wait — for example an execution that holds a lock while synchronously waiting on a child execution that wants the same lock — is not reported as a DeadlockError and resolves through the lock wait timeout instead.