Skip to main content
Version: 8 - Apfelstrudel

Secret Handling

You can use our Vault Integration to connect to an external vault service and retrieve secrets without storing them in Engine.

Use Cases

The Vault Integration offers you the possibility to authenticate to external services:

  • Authenticate yourself towards services integrated in Engine
  • Use secrets (passwords, usernames, keys, ...) in flows without "hard-coding" them
  • Interact with your Vault service (for example write secrets)

Concept

Engine offers the integration of a HashiCorp Vault (for more information see https://www.vaultproject.io/).

You can interact with a Vault using:

  • Vault Configuration: configure access to your Vault service in Engine.
  • Connections: configure which secrets will be used from your Vault service for a particular connection. When a saved connector is used, it will retrieve the secrets.
  • Connector type VAULT: provides an interface for exhaustive interaction with your Vault (read, write, update, versioning of secrets, change secret-metadata, ...).

Create a Vault Configuration

You can create a Vault configuration in the User Interface by pressing the Create button and selecting Vault config:

The buttons to create a Vault configuration

The new Vault configuration opens and you can directly modify its fields.

note

Authenticate with the Vault Token or AppRole method.

Find the fields used in the Vault configuration and their meanings in the table below:

Common fields:

FieldDescriptionExample
EnabledIf unset, Engine will not use this Vault configuration.
Auto-renewIf set, Engine will try to renew the token before it expires. Renewal will only succeed if the MAX_TTL of the token is not reached. Please refer to token renew for details.
Vault URLThe URL to your vault installationhttps://vault.example.com:8200
CA certificateA certificate to verify the identity of the vault. Only needed if the Vault installation uses a self-signed certificate.
Engine pathThe Vault engine to use, often secret or kv.

Specific for the Token authentication method:

FieldDescriptionExample
TokenA Vault access token which is used to fetch secrets.

Specific for the AppRole authentication method:

FieldDescriptionExample
AppRole PathPath for AppRole authentification method (final path for this authentification method will be /auth/<approle_path> and must be the same path where you enabled this method on your Vault)my-app-approle
Role-ID for AppRole workerRole-ID of worker. Associated with a role with permission to request wrapped SecretIDs.
Secret-ID for AppRole workerSecret-ID of worker.
Role name for AppRole runnerName of a role used by a runner, with permission to retrieve secrets from vault.my-database-runner
Role-ID for AppRole runnerRole-ID for the runner.
note

The worker's purpose is only fetching the wrapped secret-ID of a runner. This is the only permission it should be granted on the Vault. The runner's role is fetching secrets. You can have one worker configured on your Vault, but multiple runners configured with different permissions to fetch specific secrets.

Learn more about the AppRole Authentication on a HashiCorp Vault and how to set up the required Roles and Policies for your Vault: Tutorial: AppRole Pull Authentication

Use the Vault Integration

Once you have set up a vault configuration, you can use it to fetch secrets from the vault and use them in your flows, connections and stored connectors. You can also write secrets.

Read secrets

Using secrets is as simple as naming the vault configration being used and giving the path to the secret.

The format for referencing a vault secret follows this pattern: vault.secret(<vault-config-name>:<path-to-secret>.<key>)

Note that since Cloudomation Engine Version 6 the Vault engine path is now configured in the Vault config.

Using vault secrets in connections is a safe way to connect to third party systems, where secrets will not be stored or exposed within Engine.

example

Let's assume there is a vault configuration already in place and it is called 'my-vault-config'. This configures access to a key-value version 2 vault to the 'secret' Vault engine path, in which you have stored the following access credentials to an Oracle database in the path 'oracle':

user: my-user
password: my-secret-password

You can use this secret in a connection in a flow:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# use vault secrets in a connection
oracle_db_version = this.connect(
connector_type='SQLORACLE',
host='my-oracle-server',
service_name'='xe',
user='vault.secret(my-vault-config:oracle.user)',
password='vault.secret(my-vault-config:oracle.password',
execute='SELECT * FROM v$version',
).get('output_value')['result']

this.log(oracle_db_version)

return this.success('all done')

Alternatively, you can configure a Connector and map vault secrets to connector values:

example

Create a connector for the SQLORACLE connector type, call it "my-oracle-connector" and enter the following into the value field:

host: my-oracle-server
service_name: xe
user: vault.secret(my-vault-config:oracle.user)
password: vault.secret(my-vault-config:oracle.password)

You can then use this connector in your flows. The reference to the vault is already stored in the connector.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# use a connector that uses vault secrets
oracle_db_version = this.connect(
'my-oracle-connector',
execute='SELECT * FROM v$version',
).get('output_value')['result']

this.log(oracle_db_version)

return this.success('all done')

The examples above assume that the vault's key-value Vault engine is configured with version 2. You can find more information about Vault secret engines at https://www.vaultproject.io/api-docs/secret

Write Secrets

Engine can write secrets to a key-value Vault engine using the Flow-API method vault_config.write_secret().

example

Write a secret to a vault.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
system.vault_config('my-vault').write_secret(
secret_path='data/my-new-secret',
data={
'username': 'cloudomation',
'password': 'super-secret',
},
)

Use the Vault Connector Type

tip

The preferred method to read secrets is the use of a connection.

The preferred method to write secrets is the use of VaultConfig.write_secret() method.

Alternatively, you can use the vault connector type to connect to any vault and perform any operation via the vault API, similar to other connector types. However reading and writing secrets using the vault connector will result in the secrets being stored in the execution record. To use the VAULT connector type it is not required to have a Vault configuration set up and saved. It is possible to create a connection of the Vault connector type with this.connect(...) (see below). All connection parameters to the Vault have to be specified.

warning

If not properly used secrets could become exposed within your flow or executions. Use this method with caution!

example

Use the Vault connector type

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# create a secret
this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
data={
'secret-key': 'secret-value',
},
token='my-vault-token',
)

# read a secret
secret_value = this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
version=None, # read latest version
token='my-vault-token',
).get('output_value')['result']['data']['data']
assert secret_value == {'secret-key': 'secret-value'}

# destroy all versions of secret
this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
mode='delete_metadata',
token='my-vault-token',
)

return this.success('all done')