Files
Engine offers several options for working with files and file systems. This article describes the file handling functionality available to you on the Engine platform.
Use Cases
You can use Engine file functionality to
- retreive files from a remote system and store them in Engine
- transfer files from Engine to remote systems
- send files as attachments of emails
- use files as templates for mails or messages
Concept
You can store binary content as files in Engine. Files can be accessed using the flow API. Several connectors directly read or write Engine files.
User Interface
The User Interface allows to upload and download files. Text files can be edited directly in the User Interface.
Upload files
- Click the "+ Create" button in the left hand menu and select "Upload file".
Buttons to upload files
- In the dialog choose one or several files and confirm.
- The upload progress is shown in the top bar.
The upload progress bar
- Clicking on a file in the upload progress panel will open the file in the User Interface.
Download files
Click the "Download" button next to any file.
The download button
Flow API
The flow API provides methods to read and write file content as bytes, base64 string, or decoded as utf-8 string.
Create, write, read, list, and delete files
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# store a text (utf8 string)
system.file('my-file.txt').save_text_content('text content')
# read as base64
assert system.file('my-file.txt').get_base64_content() == 'dGV4dCBjb250ZW50'
# store bytes data
system.file('my-file.dat').save_bytes_content(b'bytes data \xc3\xa4\xc3\xb6\xc3\xbc')
# read as text (utf8 string)
assert system.file('my-file.dat').get_text_content() == 'bytes data äöü'
# store a base64 encoded string
system.file('my-file.ext').save_base64_content('Q2xvdWRvbWF0aW9u')
# read as bytes
assert system.file('my-file.ext').get_bytes_content() == b'Cloudomation'
# list files
for file_ in system.files():
this.log(file_.get('name', 'size_bytes'))
# clean up
system.file('my-file.txt').delete()
system.file('my-file.dat').delete()
system.file('my-file.ext').delete()
return this.success('all done')
Connectors accessing files
SCP
The SCP connector allows to copy files using the SCP protocol.
Read and write a file using the SCP connector.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# copy a file from a remote SCP host to Engine
this.connect(
'my-web-server',
src='/var/log/apache2/access.log',
dst='cloudomation:apache-access.log',
)
# access the file content
content = system.file('apache-access.log').get_text_content()
# copy a file from Engine to a remote SCP host
this.connect(
'my-web-server',
src='cloudomation:index.html',
dst='/var/www/html/index.html',
)
return this.success('all done')
SMB
The SMB connector allows to copy files using the SMB protocol.
Read and write a file using the SMB connector.
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# copy a file from a fileshare to Engine
this.connect(
'my-windows-host',
src='share-name\\path\\to\\report.xlsx',
dst='cloudomation:report.xlsx',
)
# copy a file from Engine to a fileshare
this.connect(
'my-windows-host',
src='cloudomation:processing.log',
dst='report-share\\processing.log',
)
return this.success('all done')
GIT
The GIT connector allows to fetch files from a repository and store them in Engine
Fetch files from git and store them in Engine
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
this.connect(
connector_type='GIT',
command='get',
repository_url='https://example.com/path/to/repo.git',
ref='develop',
files_path='files-from-repository'
)
# list files which were fetched
for file_ in system.files(
filter_={
'field': 'name',
'op': 'like',
'value': 'files-from-repository/%',
},
):
this.log(file_.get('name', 'size_bytes'))
# clean up
file_.delete()
return this.success('all done')
IMAP
The IMAP connector can store email attachments in Engine
Fetch an email and store the attachments in Engine
import datetime
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
# search for todays messages with "report" in the subject line
today = datetime.date.today().strftime('%d-%b-%Y')
message_ids = this.connect(
'my-imap-server',
name='search for mails',
command='search',
folder='INBOX',
criteria=[
'SUBJECT', 'report',
'SINCE', today,
],
).get('output_value')['result']
messages = this.connect(
'my-imap-server',
name='fetch mails',
command='fetch',
message_set=message_ids,
# attachments are stored as `{message-id}-{attachment-file-name}`
store_attachments=True,
).get('output_value')['result']
# list all attachments which were fetched
for message_id in message_ids:
for file_ in system.files(
filter_={
'field': 'name',
'op': 'like',
'value': f'{message_id}-%',
},
):
this.log(f'attachment of message {message_id}: {file_.get("name")})
return this.success('all done')
SMTP
The SMTP connector allows to send mails with attachments from Engine files.
Send a mail with attachment
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
this.connect(
'my-smtp-server',
from='no-reply@example.com',
to='user@example.com',
subject='monthly report',
text='the monthly processing report is attached',
attachments=[
# attach files from the Engine files resource
'cloudomation:report.csv',
'cloudomation:report.sig',
# attach files from an URL
'https://cloudomation.com/wp-content/uploads/2020/11/1-1.jpg'
],
)
return this.success('all done')
REST
The REST connector allows to make multipart POST requests with file parts from Engine.
Make a multipart POST request containing a file from Engine
import flow_api
def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
this.connect(
connector_type='REST',
url='https://httpbin.org/post',
method='POST',
multipart={
'parts': [
{
'name': 'string-field',
'value': 'spam & eggs',
},
{
'name': 'file',
'file': 'report.txt',
},
],
},
)
return this.success('all done')