SSH Tunneling
Concept
You can use SSH tunneling to safely connect to external systems that are not accessible via the public internet. Once a tunnel is established, you can use Engine connectors to interact with the external system through the tunnel.
How it works
Let's say you want to connect to a PostgreSQL database on a remote host. The database runs on port 5432 of the remote host. The host only allows SSH connections so you cannot directly connect to the database, even if you have the credentials.
To connect to the database, you can create an SSH tunnel to the remote host via the SSH connector. The connector will open a connection to the remote host via SSH and will then tunnel the database port (5432) to your local machine on a random port. The host and port are saved in the connection execution's output value.
Now you can use an SQLPG connector to connect to the database through the tunnel by using the host and port from the output value of the SSH connection execution.
To close the tunnel, simply cancel the SSH connection execution.
Example
First, create an SSH connector. Then configure it with the following parameters:
Be mindful when hardcoding credentials like SSH keys and passwords in connectors and flow scripts. To securely manage credentials, use our integrated secrets managers.
host: <remote_host>
authentication:
authentication_method: ssh_key
username: <your_username>
ssh_key: <your_ssh_private_key>
mode:
mode_name: 'forward_port'
destination_host: 'localhost' # the host of the database on the remote host
destination_port: 5432 # the port of the database on the remote host
# other connector parameters
Then run the SSH connector and connect to the database through the tunnel:
import contextlib
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# create the tunnel
port_forwarding_task = this.connect(
'<your_ssh_connector_name>',
wait=False,
)
try:
# wait for the tunnel to be created
for _ in range(5):
this.sleep(1)
tunnel = port_forwarding_task.get('output_value')
if tunnel is not None:
break
else:
return this.error('no tunnel')
# connect to the database through the tunnel
this.connect(
connector_type='SQLPG',
name='test PostgreSQL connection',
**tunnel,
database='<your_database_name>',
authentication={
'authentication_method': 'username_password',
'username': '<your_username>',
'password': '<your_password>',
},
mode={
'mode_name': 'fetchval',
'query': '<your_query>',
},
).get('output_value')['result']
finally:
# cancel the tunnel
port_forwarding_task.cancel()
with contextlib.suppress(flow_api.DependencyFailedError):
port_forwarding_task.wait()
return this.success('all done')
Jump hosts (ProxyJump)
The SSH, SCP and SFTP connectors accept an optional jump input to reach a target that is only accessible through one or more intermediate SSH jump hosts (bastions). Each hop is connected through the previous one using SSH's native tunneling, so a single connector call reaches the target directly — without a separate forward_port tunnel or a local listener.
jump is an ordered list of hops, from the hop closest to Cloudomation Engine (first) to the hop closest to the target (last). Omitting jump, or passing an empty list, connects directly to the target.
Each hop is one of two modes:
connector— reuse an existing SSH, SCP or SFTP connector as the jump host. Its stored host, port, credentials and hostkey are used. This keeps jump-host credentials defined and rotated in a single place.inline— define the jump host'shost,port,authenticationandhostkeydirectly on the hop.
Be mindful when hardcoding credentials like SSH keys and passwords in connectors and flow scripts. To securely manage credentials, use our integrated secrets managers.
Example: copy a file through a jump host
Copy a file from a target host that is only reachable through a bastion, defining the bastion inline:
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
this.connect(
connector_type='SCP',
name='copy report via bastion',
host='<target_host>', # only reachable from the jump host
authentication={
'authentication_method': 'ssh_key',
'username': '<target_username>',
'ssh_key': '<target_ssh_private_key>',
},
hostkey={
'hostkey_mode': 'known_host',
'known_host': '<target_known_host_line>',
},
jump=[
{
'jump_mode': 'inline',
'host': '<bastion_host>',
'port': {'port_mode': 'port_number', 'port_number': 22},
'authentication': {
'authentication_method': 'ssh_key',
'username': '<bastion_username>',
'ssh_key': '<bastion_ssh_private_key>',
},
'hostkey': {
'hostkey_mode': 'known_host',
'known_host': '<bastion_known_host_line>',
},
},
],
mode={
'mode_name': 'copy_file_to_engine',
'source_file_name': '/data/report.csv',
'destination_file_name': 'report.csv',
'destination_location': {'location_mode': 'inherit_from_execution'},
},
)
return this.success('all done')
To reuse an existing connector as the jump host instead of defining it inline, use connector mode and reference the connector by id:
jump=[
{'jump_mode': 'connector', 'connector_id': '<jump connector id>'},
],
Multiple hops
Chain several jump hosts by listing more hops, ordered from Cloudomation Engine to the target:
jump=[
{'jump_mode': 'connector', 'connector_id': '<first bastion>'}, # closest to Engine
{'jump_mode': 'connector', 'connector_id': '<second bastion>'}, # closest to the target
],
The jump input works the same way for the SSH and SFTP connectors.