Secret Handling

You can retrieve secrets without storing them in Cloudomation Engine, by using either (or both) of two integrated secret manager services:

• HashiCorp Vault (for more information see https://www.vaultproject.io/)
• Devolutions Server https://devolutions.net/server/

Use Cases

The above integrations offer you the possibility to authenticate to external services:

• Authenticate yourself towards services integrated in Engine
• Use secrets (passwords, usernames, keys, ...) in flows without "hard-coding" them
• Interact with your secret manager service e.g. write secrets (only for HashiCorp Vault)

Concept

You can interact with a secret manager service via:

• Vault Configuration/Devolutions Configuration: configure access to your HashiCorp Vault or Devolutions Server in Engine.
• Connections: configure which secrets will be used from your secret manager service for a particular connection. When a saved connector is used, it will retrieve the secrets.
• Connector type VAULT: provides an interface for exhaustive interaction with your Vault (read, write, update, versioning of secrets, change secret-metadata, ...).

warning

When a key in the input value of a connection is a secret, there are two ways the secret is resolved:

1. The key contains only a reference to a secret e.g.

   {'users': 'vault.secret(my-vault-config:oracle.user)'}:

   The key is resolved with the secret, preserving the data type of the secret. E.g. if the secret is a list, the resolved value will be a list as well:

   {'users': ['user1', 'user2']}

2. The key does not only contain a reference to a secret e.g.

   {'users': ['vault.secret(my-vault-config:oracle.user)', 'some_other_element']}:

   The key is resolved with the secret, but the data type of the secret will be converted to a string. E.g. if the secret is a list, the resolved value will be a string:

   {'users': ['[\'user1\', \'user2\']', 'some_other_element']}

HashiCorp Vault

Create a Vault Configuration

You can create a Vault configuration in the User Interface by pressing the Create button then selecting Configuration and Vault config:

The buttons to create a Vault configuration

The new Vault configuration opens and you can directly modify its fields.

note

Authenticate with the Vault Token or Username & Password method.

Find the fields used in the Vault configuration and their meanings in the table below:

Common fields:

Field	Description	Example
Enabled	If unset, Engine will not use this Vault configuration.
Auto-renew	If set, Engine will try to renew the token before it expires. Renewal will only succeed if the MAX_TTL of the token is not reached. Please refer to token renew for details.
Vault URL	The URL to your vault installation	https://vault.example.com:8200
Engine path	The Vault engine to use, often secret or kv.
CA certificate	A certificate to verify the identity of the vault. Only needed if the Vault installation uses a self-signed certificate.
Check hostname	If set, the hostname of the server is checked against the CA certificate.
Verify SSL	Verify the server's SSL certificate. Strongly recommended. Can be disabled if using a self-signed certificate.
Client certificate	An optional client certificate used to authenticate the SSL transport.
Client certificate key	The key of the client certificate used to authenticate the SSL transport.

Specific for the Token authentication method:

Field	Description	Example
Token	A Vault access token which is used to fetch secrets.

Specific for the Username & Password authentication method:

Field	Description	Example
Username	The username of the vault user.
Password	The password of the vault user.

Use the Vault Integration

Once you have set up a vault configuration, you can use it to fetch secrets from the vault and use them in your flows, connections and stored connectors. You can also write secrets.

Read secrets

Using secrets is as simple as naming the vault configration being used and giving the path to the secret.

The format for referencing a vault secret follows this pattern: vault.secret(<vault-config-name>:<path-to-secret>.<key>)

Note that since Cloudomation Engine Version 6 the Vault engine path is now configured in the Vault config.

Using vault secrets in connections is a safe way to connect to third party systems, where secrets will not be stored or exposed within Engine.

example

Let's assume there is a vault configuration already in place and it is called 'my-vault-config'. This configures access to a key-value version 2 vault to the 'secret' Vault engine path, in which you have stored the following access credentials to an Oracle database in the path 'oracle':

user: my-user
password: my-secret-password

You can use this secret in a connection in a flow:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # use vault secrets in a connection
    oracle_db_version = this.connect(
        connector_type='SQLORACLE',
        host='my-oracle-server',
        service_name'='xe',
        user='vault.secret(my-vault-config:oracle.user)',
        password='vault.secret(my-vault-config:oracle.password)',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

Alternatively, you can configure a Connector and map vault secrets to connector values:

example

Create a connector for the SQLORACLE connector type, call it "my-oracle-connector" and enter the following into the value field:

host: my-oracle-server
service_name: xe
user: vault.secret(my-vault-config:oracle.user)
password: vault.secret(my-vault-config:oracle.password)

You can then use this connector in your flows. The reference to the vault is already stored in the connector.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
  # use a connector that uses vault secrets
    oracle_db_version = this.connect(
        'my-oracle-connector',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

The examples above assume that the vault's key-value Vault engine is configured with version 2. You can find more information about Vault secret engines at https://www.vaultproject.io/api-docs/secret

Write Secrets

Engine can write secrets to a key-value Vault engine using the Flow-API method vault_config.write_secret().

example

Write a secret to a vault.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    system.vault_config('my-vault').write_secret(
        engine_path='kv',
        secret_path='data/my-new-secret',
        data={
            'username': 'cloudomation',
            'password': 'super-secret',
        },
    )

Use the Vault Connector Type

tip

The preferred method to read secrets is the use of a connection.

The preferred method to write secrets is the use of VaultConfig.write_secret() method.

Alternatively, you can use the vault connector type to connect to any vault and perform any operation via the vault API, similar to other connector types. However reading and writing secrets using the vault connector will result in the secrets being stored in the execution record. To use the VAULT connector type it is not required to have a Vault configuration set up and saved. It is possible to create a connection of the Vault connector type with this.connect(...) (see below). All connection parameters to the Vault have to be specified.

warning

If not properly used secrets could become exposed within your flow or executions. Use this method with caution!

example

Use the Vault connector type

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # create a secret
    this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        mode={
            'mode_name': 'upsert',
            'data' : {
                'secret-key': 'secret-value',
            },
        },
        token='my-vault-token',
    )

    # read a secret
    secret_value = this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        version=None,  # read latest version
        token='my-vault-token',
    ).get('output_value')['result']['data']['data']
    assert secret_value == {'secret-key': 'secret-value'}

    # destroy all versions of secret
    this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        mode='delete_metadata',
        token='my-vault-token',
    )

    return this.success('all done')

Devolutions Server

Create a Devolutions Configuration

You can create a Devolutions configuration in the User Interface by pressing the Create button then selecting Configuration and Devolutions config:

The buttons to create a devolutions configuration

The new devolutions configuration opens and you can directly modify its fields.

note

You can authenticate with the combination of App key & App secret.

Find the fields used in the devolutions configuration and their meanings in the table below:

Field	Description	Example
Enabled	If unset, Engine will not use this Vault configuration.
Devolutions URL	The URL to your Devolutions Server installation	https://devolutions.example.com/dvls
CA certificate	A certificate to verify the identity of the vault. Only needed if the Vault installation uses a self-signed certificate.
Verify SSL	Verify the server's SSL certificate. Strongly recommended. Can be disabled if using a self-signed certificate.
Vault ID	The ID of the Devolutions vault that this configuration should point to.	00000000-0000-0000-0000-000000000000
App key	The App key to use for authentication.
App secret	The App secret to use for authentication.

Use the Devolutions Integration

Once you have set up a devolutions configuration, you can use it to fetch secrets from a devolutions vault and use them in your flows, connections and stored connectors.

You can reference a devolutions secret either by its fullpath (path and name) and secret type, or by its uuid.

The format for referencing a devolutions secret can be either of these patterns:

1. referencing by fullpath and secret type:

   devolutions.secret(<devolutions-config-name>:<fullpath>@<secret_type>.<secret_key>)

2. referencing by uuid:

   devolutions.secret(<devolutions-config-name>:<uuid>.<secret_key>)

Using devolutions secrets in connections is a safe way to connect to third party systems, where secrets will not be stored or exposed within Engine.

example

Let's assume there is a devolutions configuration already in place and it is called my-devolutions-config. This configures access to a key-value devolutions vault in which you have stored the following access information of type Credentials to an Oracle database in the fullpath oracle:

user: my-user
password: my-secret-password

You can use this secret in a connection in a flow:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # use devolutions secrets in a connection
    oracle_db_version = this.connect(
        connector_type='SQLORACLE',
        host='my-oracle-server',
        service_name'='xe',
        user='devolutions.secret(my-devolutions-config:oracle@Credentials.user)',
        password='devolutions.secret(my-devolutions-config:oracle@Credentials.password)',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

Alternatively, you can configure a Connector and map devolutions secrets to connector values:

example

Create a connector for the SQLORACLE connector type, call it "my-oracle-connector" and enter the following into the value field:

host: my-oracle-server
service_name: xe
user: devolutions.secret(my-devolutions-config:oracle@Credentials.user)
password: devolutions.secret(my-devolutions-config:oracle@Credentials.password)

You can then use this connector in your flows. The reference to the devolutions vault is already stored in the connector.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
  # use a connector that uses devolutions secrets
    oracle_db_version = this.connect(
        'my-oracle-connector',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

Now assume, that you create a folder named database-credentials in your devolutions vault, and move the secret oracle into it. The fullpath becomes database-credentials/oracle.

example

To reference the secret in a folder, use the following format:

host: my-oracle-server
service_name: xe
user: devolutions.secret(my-devolutions-config:database-credentials/oracle@Credentials.user)
password: devolutions.secret(my-devolutions-config:database-credentials/oracle@Credentials.password)

Finally, you can referernce secrets by their UUID. This comes especially handy, when there are multiple secrets in your devolutions vault that have the same fullpath and secret type. Let's assume that the uuid of the secret oracle is cd91da1b-9e77-48f0-bb30-41e4144b8a18

example

To reference the secret by UUID, use the following format:

host: my-oracle-server
service_name: xe
user: devolutions.secret(my-devolutions-config:cd91da1b-9e77-48f0-bb30-41e4144b8a18.user)
password: devolutions.secret(my-devolutions-config:cd91da1b-9e77-48f0-bb30-41e4144b8a18.password)

Combining the two integrations

If you use both HashiCorp Vault and Devolutions Server, you can also combine their usage within the same connection.

example

Let's say that you use HashiCorp Vault to store credentials and Devolutions Server to store Oracle queries.

The configurations are called 'my-vault-config' and 'my-devolutions-config' and the path is in both cases 'oracle'. The secret type in devolutions is 'Credentials'.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # use devolutions secrets in a connection
    oracle_db_version = this.connect(
        connector_type='SQLORACLE',
        host='my-oracle-server',
        service_name'='xe',
        user='vault.secret(my-vault-config:oracle.user)',
        password='vault.secret(my-vault-config:oracle.password)',
        execute='devolutions.secret(my-devolutions-config:oracle@Credentials.execute)',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')