Secret Handling
You can use our Vault Integration to connect to an external vault service and retrieve secrets without storing them in Engine.
Use Cases
The Vault Integration offers you the possibility to authenticate to external services:
- Authenticate yourself towards services integrated in Engine
- Use secrets (passwords, usernames, keys, ...) in flows without "hard-coding" them
- Interact with your Vault service (for example write secrets)
Concept
Engine offers the integration of a HashiCorp Vault (for more information see https://www.vaultproject.io/).
You can interact with a Vault using:
- Vault Configuration: configure access to your Vault service in Engine.
- Connections: configure which secrets will be used from your Vault service for a particular connection. When a saved connector is used, it will retrieve the secrets.
- Connector type VAULT: provides an interface for exhaustive interaction with your Vault (read, write, update, versioning of secrets, change secret-metadata, ...).
Create a Vault Configuration
You can create a Vault configuration in the User Interface by pressing the Create button and selecting Vault config:
The buttons to create a Vault configuration
The new Vault configuration opens and you can directly modify its fields.
Authenticate with the Vault Token or AppRole method.
Find the fields used in the Vault configuration and their meanings in the table below:
Common fields:
| Field | Description | Example |
|---|---|---|
| Enabled | If unset, Engine will not use this Vault configuration. | |
| Auto-renew | If set, Engine will try to renew the token before it expires. Renewal will only succeed if the MAX_TTL of the token is not reached. Please refer to token renew for details. | |
| Vault URL | The URL to your vault installation | https://vault.example.com:8200 |
| CA certificate | A certificate to verify the identity of the vault. Only needed if the Vault installation uses a self-signed certificate. | |
| Engine path | The Vault engine to use, often secret or kv. |
Specific for the Token authentication method:
| Field | Description | Example |
|---|---|---|
| Token | A Vault access token which is used to fetch secrets. |
Specific for the AppRole authentication method:
| Field | Description | Example |
|---|---|---|
| AppRole Path | Path for AppRole authentification method (final path for this authentification method will be /auth/<approle_path> and must be the same path where you enabled this method on your Vault) | my-app-approle |
| Role-ID for AppRole worker | Role-ID of worker. Associated with a role with permission to request wrapped SecretIDs. | |
| Secret-ID for AppRole worker | Secret-ID of worker. | |
| Role name for AppRole runner | Name of a role used by a runner, with permission to retrieve secrets from vault. | my-database-runner |
| Role-ID for AppRole runner | Role-ID for the runner. |
The worker's purpose is only fetching the wrapped secret-ID of a runner. This is the only permission it should be granted on the Vault. The runner's role is fetching secrets. You can have one worker configured on your Vault, but multiple runners configured with different permissions to fetch specific secrets.
Learn more about the AppRole Authentication on a HashiCorp Vault and how to set up the required Roles and Policies for your Vault: Tutorial: AppRole Pull Authentication
Use the Vault Integration
Once you have set up a vault configuration, you can use it to fetch secrets from the vault and use them in your flows, connections and stored connectors. You can also write secrets.
Read secrets
Using secrets is as simple as naming the vault configration being used and giving the path to the secret.
The format for referencing a vault secret follows this pattern:
vault.secret(<vault-config-name>:<path-to-secret>.<key>)
Note that since Cloudomation Engine Version 6 the Vault engine path is now configured in the Vault config.
Using vault secrets in connections is a safe way to connect to third party systems, where secrets will not be stored or exposed within Engine.
Let's assume there is a vault configuration already in place and it is called 'my-vault-config'. This configures access to a key-value version 2 vault to the 'secret' Vault engine path, in which you have stored the following access credentials to an Oracle database in the path 'oracle':
user: my-user
password: my-secret-password
You can use this secret in a connection in a flow:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# use vault secrets in a connection
oracle_db_version = this.connect(
connector_type='SQLORACLE',
host='my-oracle-server',
service_name'='xe',
user='vault.secret(my-vault-config:oracle.user)',
password='vault.secret(my-vault-config:oracle.password',
execute='SELECT * FROM v$version',
).get('output_value')['result']
this.log(oracle_db_version)
return this.success('all done')
Alternatively, you can configure a Connector and map vault secrets to connector values:
Create a connector for the SQLORACLE connector type, call it "my-oracle-connector" and enter the following into the value field:
host: my-oracle-server
service_name: xe
user: vault.secret(my-vault-config:oracle.user)
password: vault.secret(my-vault-config:oracle.password)
You can then use this connector in your flows. The reference to the vault is already stored in the connector.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# use a connector that uses vault secrets
oracle_db_version = this.connect(
'my-oracle-connector',
execute='SELECT * FROM v$version',
).get('output_value')['result']
this.log(oracle_db_version)
return this.success('all done')
The examples above assume that the vault's key-value Vault engine is configured with version 2. You can find more information about Vault secret engines at https://www.vaultproject.io/api-docs/secret
Write Secrets
Engine can write secrets to a key-value Vault engine using the Flow-API method vault_config.write_secret().
Write a secret to a vault.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
system.vault_config('my-vault').write_secret(
secret_path='data/my-new-secret',
data={
'username': 'cloudomation',
'password': 'super-secret',
},
)
Use the Vault Connector Type
The preferred method to read secrets is the use of a connection.
The preferred method to write secrets is the use of VaultConfig.write_secret() method.
Alternatively, you can use the vault connector type to connect to any vault and perform any operation via the vault API, similar to other connector types. However reading and writing secrets using the vault connector will result in the secrets being stored in the execution record.
To use the VAULT connector type it is not required to have a Vault configuration set up and saved.
It is possible to create a connection of the Vault connector type with this.connect(...) (see below).
All connection parameters to the Vault have to be specified.
If not properly used secrets could become exposed within your flow or executions. Use this method with caution!
Use the Vault connector type
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# create a secret
this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
data={
'secret-key': 'secret-value',
},
token='my-vault-token',
)
# read a secret
secret_value = this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
version=None, # read latest version
token='my-vault-token',
).get('output_value')['result']['data']['data']
assert secret_value == {'secret-key': 'secret-value'}
# destroy all versions of secret
this.connect(
'VAULT',
host='https://my-vault-host:8200',
engine_path='kv',
secret_path='data/my-secret',
mode='delete_metadata',
token='my-vault-token',
)
return this.success('all done')