Secret Handling

You can use our Vault Integration to connect to an external vault service and retrieve secrets without storing them in Cloudomation Engine.

Use Cases

The Vault Integration offers you the possibility to authenticate to external services:

• Authenticate yourself towards services integrated in Engine
• Use secrets (passwords, usernames, keys, ...) in flows without "hard-coding" them
• Interact with your Vault service (for example write secrets)

Concept

Engine offers the integration of a HashiCorp Vault (for more information see https://www.vaultproject.io/).

You can interact with a Vault using:

• Vault Configuration: configure access to your Vault service in Engine.
• Connections: configure which secrets will be used from your Vault service for a particular connection. When a saved connector is used, it will retrieve the secrets.
• Connector type VAULT: provides an interface for exhaustive interaction with your Vault (read, write, update, versioning of secrets, change secret-metadata, ...).

Create a Vault Configuration

You can create a Vault configuration in the User Interface by pressing the Create button and selecting Vault config:

The buttons to create a Vault configuration

The new Vault configuration opens and you can directly modify its fields. Find the fields used in the Vault configuration and their meanings in the table below:

Field	Description	Example
Enabled	If unset, Engine will not use this Vault configuration.
Auto-renew	If set, Engine will try to renew the token before it expires. Renewal will only succeed if the MAX_TTL of the token is not reached. Please refer to token renew for details.
Vault URL	The URL to your vault installation	https://vault.example.com:8200
CA certificate	A certificate to verify the identity of the vault. Only needed if the Vault installation uses a self-signed certificate.
Engine path	The Vault engine to use, often secret or kv.
Token	A Vault access token which is used to fetch secrets.

Use the Vault Integration

Once you have set up a vault configuration, you can use it to fetch secrets from the vault and use them in your flows, connections and stored connectors. You can also write secrets.

Read secrets

Using secrets is as simple as naming the vault configration being used and giving the path to the secret.

The format for referencing a vault secret follows this pattern: vault.secret(<vault-config-name>:<path-to-secret>.<key>)

Note that since Cloudomation Engine Version 6 the Vault engine path is now configured in the Vault config.

Using vault secrets in connections is a safe way to connect to third party systems, where secrets will not be stored or exposed within Engine.

example

Let's assume there is a vault configuration already in place and it is called 'my-vault-config'. This configures access to a key-value version 2 vault to the 'secret' Vault engine path, in which you have stored the following access credentials to an Oracle database in the path 'oracle':

user: my-user
password: my-secret-password

You can use this secret in a connection in a flow:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # use vault secrets in a connection
    oracle_db_version = this.connect(
        connector_type='SQLORACLE',
        host='my-oracle-server',
        service_name'='xe',
        user='vault.secret(my-vault-config:oracle.user)',
        password='vault.secret(my-vault-config:oracle.password)',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

Alternatively, you can configure a Connector and map vault secrets to connector values:

example

Create a connector for the SQLORACLE connector type, call it "my-oracle-connector" and enter the following into the value field:

host: my-oracle-server
service_name: xe
user: vault.secret(my-vault-config:oracle.user)
password: vault.secret(my-vault-config:oracle.password)

You can then use this connector in your flows. The reference to the vault is already stored in the connector.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
  # use a connector that uses vault secrets
    oracle_db_version = this.connect(
        'my-oracle-connector',
        execute='SELECT * FROM v$version',
    ).get('output_value')['result']
  
    this.log(oracle_db_version)
  
    return this.success('all done')

The examples above assume that the vault's key-value Vault engine is configured with version 2. You can find more information about Vault secret engines at https://www.vaultproject.io/api-docs/secret

Write Secrets

Engine can write secrets to a key-value Vault engine using the Flow-API method vault_config.write_secret().

example

Write a secret to a vault.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    system.vault_config('my-vault').write_secret(
        engine_path='kv',
        secret_path='data/my-new-secret',
        data={
            'username': 'cloudomation',
            'password': 'super-secret',
        },
    )

Use the Vault Connector Type

tip

The preferred method to read secrets is the use of a connection.

The preferred method to write secrets is the use of VaultConfig.write_secret() method.

Alternatively, you can use the vault connector type to connect to any vault and perform any operation via the vault API, similar to other connector types. However reading and writing secrets using the vault connector will result in the secrets being stored in the execution record. To use the VAULT connector type it is not required to have a Vault configuration set up and saved. It is possible to create a connection of the Vault connector type with this.connect(...) (see below). All connection parameters to the Vault have to be specified.

warning

If not properly used secrets could become exposed within your flow or executions. Use this method with caution!

example

Use the Vault connector type

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # create a secret
    this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        mode={
            'mode_name': 'upsert',
            'data' : {
                'secret-key': 'secret-value',
            },
        },
        token='my-vault-token',
    )

    # read a secret
    secret_value = this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        version=None,  # read latest version
        token='my-vault-token',
    ).get('output_value')['result']['data']['data']
    assert secret_value == {'secret-key': 'secret-value'}

    # destroy all versions of secret
    this.connect(
        'VAULT',
        host='https://my-vault-host:8200',
        engine_path='kv',
        secret_path='data/my-secret',
        mode='delete_metadata',
        token='my-vault-token',
    )

    return this.success('all done')