SSH Tunneling

Concept

You can use SSH tunneling to safely connect to external systems that are not accessible via the public internet. Once a tunnel is established, you can use Engine connectors to interact with the external system through the tunnel.

How it works

Let's say you want to connect to a PostgreSQL database on a remote host. The database runs on port 5432 of the remote host. The host only allows SSH connections so you cannot directly connect to the database, even if you have the credentials.

To connect to the database, you can create an SSH tunnel to the remote host via the SSH connector. The connector will open a connection to the remote host via SSH and will then tunnel the database port (5432) to your local machine on a random port. The host and port are saved in the connection execution's output value.

Now you can use an SQLPG connector to connect to the database through the tunnel by using the host and port from the output value of the SSH connection execution.

To close the tunnel, simply cancel the SSH connection execution.

Example

First, create an SSH connector. Then configure it with the following parameters:

warning

Be mindful when hardcoding credentials like SSH keys and passwords in connectors and flow scripts. To securely manage credentials, use our integrated secrets managers.

host: <remote_host>
authentication:
    authentication_method: ssh_key
    username: <your_username>
    ssh_key: <your_ssh_private_key>
mode:
    mode_name: 'forward_port'
    destination_host: 'localhost' # the host of the database on the remote host
    destination_port: 5432 # the port of the database on the remote host
# other connector parameters

Then run the SSH connector and connect to the database through the tunnel:

import contextlib
import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # create the tunnel
    port_forwarding_task = this.connect(
        '<your_ssh_connector_name>',
        wait=False,
    )
    try:
        # wait for the tunnel to be created
        for _ in range(5):
            this.sleep(1)
            tunnel = port_forwarding_task.get('output_value')
            if tunnel is not None:
                break
        else:
            return this.error('no tunnel')
        # connect to the database through the tunnel
        this.connect(
            connector_type='SQLPG',
            name='test PostgreSQL connection',
            **tunnel,
            database='<your_database_name>',
            authentication={
                'authentication_method': 'username_password',
                'username': '<your_username>',
                'password': '<your_password>',
            },
            mode={
                'mode_name': 'fetchval',
                'query': '<your_query>',
            },
        ).get('output_value')['result']
    finally:
        # cancel the tunnel
        port_forwarding_task.cancel()
        with contextlib.suppress(flow_api.DependencyFailedError):
            port_forwarding_task.wait()
    return this.success('all done')

Learn More

Connectors

Connector Types

Secret Handling