Git Integration

Using the git integration it is possible to synchronise the content of a git repository with Engine.

Use Cases

Generally it is recommended to store all your Engine resources in git. There are plenty of benefits:

• Use of your favourite offline editor
• Version history of all changes to your automation logic
• Collaboration between users easily possible
• An additional backup of your automation logic
• Branching, merging, rollback, ...

Concept

With git integration configured, all saved changes to resources in Engine are automatically synced to a remote repository that is provided by Engine as part of the workspace. This means, that whenever you change a flow script, enable or disable a schedule, rename a setting etc... a commit is generated and pushed upon saving. The author of the commit is your Engine user.

On the other hand, you can also make changes to resources in your local repository and upon committing and pushing, these changes will be reflected in Engine.

Scope of the synchronisation

You can set up the synchronisation for projects, bundles or the workspace. Only resources within the given project or bundle will be tracked. The git integration for the workspace tracks resources that are not part of a project or a bundle (e.g. users). Activities (such as executions or messages) are not tracked by the git integration.

important

Activities (e.g. executions or messages) are not tracked by the git integration.

Folder structure

Tracked resources can be found in the designated cloudomation-resources folder of the repository. Everything outside of this folder will be ignored by the git integration and can be used for your own purposes e.g. readme files, templates, etc.

note

The resources you will find in cloudomation-resources have the file pattern like any other Cloudomation export i.e. a resource can be represented by of multiple files. For example a flow called my-flow is exported as my-flow.flow-description.txt and my-flow.flow-script.py.

You can read more about Cloudomation export file formats here.

Initial Setup

Per default, the git integration is not set up. To enable the integration navigate to the project/bundle/workspace that you want to synchronize.

Setting up the remote repository

Click on the git actions button to trigger the dialogue for creating a new remote repository, then choose the name of your branch (e.g. develop), write it into the field "Branch name", and click "Create git repository".

The git actions button.

The dialogue for creating a repository.

When a project/bundle/workspace is git enabled, a commit button is shown.

Also the git actions button is marked with a green dot and offers options to manage the git integration.

From now on every change you make, that falls into the scope of the synchronization, will be tracked and you can commit them into the remote repository.

Setting up the local repository

To set up the local repository you can clone the remote repository. You can find the repository URL for cloning by clicking on the git actions button and choosing "Manage git repository". There you will see the "Git URL" in the dialogue window.

The URL for cloning the repository.

Assigning your public key

Since the cloning uses the SSH transfer protocol, make sure that your Engine user has your SSH public key assigned.

You can see how to check this as described in Adding SSH Public Keys. If there are no public keys listed you need to add the content of your public key.

Cloning the remote repository

Once your user has your public SSH key assigned you can create the local repository by cloning the remote.

This can be done on the command line or you can use the Repository Cloning feature of your offline editor (if it has one).

tip

The git URL is automatically put into quotes if the URL has spaces or special characters, so the command line handles it correctly.

Tools like VSCode or TortoiseGit handle quotes differently and might run into an error like fatal: protocol ''ssh' is not supported. In this case remove the quotes from the URL after copying it.

Example: Cloning via command line

1. open a command line in the folder where you want to create the local repository
2. execute the git clone command with the git URL as parameter. E.g.:

git clone "ssh://git@localhost:2022/git/project/Default project.git" Default

Developing with the git integration

Once you have your local repository you can use git in both directions, synching changes from Engine to the local repository, or from the local repository to Engine.

Synching from the local repository to Engine is done in the same way as you would work with any git repository. You can make changes locally, commit and push them to the remote repository. These changes will then be reflected in Engine.

Committing in Engine

Whether your changes to a resource in a git enabled project/bundle/workspace are automatically committed upon saving, depends on the workspace seting GIT_AUTOCOMMIT_ENABLED

The save button, when GIT_AUTOCOMMIT_ENABLED is set to True

If automatic committing is not enabled, you can manually commit changes, individually or together with other changes, using dedicated git actions in Engine.

warning

If the git integration is enabled, it is best practice to not accumulate too many uncommited changes. Once you have finished working on a change, make sure to commit it or if you don't want to keep the changes, reset them to the status in git.

In the UI

Let's add 2 new flows to the project: flow-1 and flow-2. You can commit these changes individually, by navigating to the new flows and clicking on their respective commit buttons. This will open a dialogue that lets you edit the commit message, and also lets you switch to bulk commit mode.

The dialogue for committing a single change

Instead of committing, let's switch to bulk commit.

The dialogue lets you choose which resources you want to commit and also lets you generate a commit message. When you are ready, click on "Bulk commit".

The dialogue for committing a multiple changes

Now with the selection and a generated commit message

The message to signal a successful commit

tip

Clicking the commit button on a resource open a dialogue to commit changes to that resource only. You can access bulk commit either from this dialogue window, or by clicking on the ellipsis next to the commit button and choosing "Bulk commit".

In the flow_api

You can commit changes to a resource with the commit method. This will use a generated commit message.

example

Modifying the description of a flow and committing it:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    flow = system.flow('my-flow')
    flow.save(
        description='changed description',
    )
    flow.commit()

    return this.success('all done')

note

Using the commit method without any parameters will commit all changes to a resource.

You can modify the commit message by supplying the message argument to the commit method. You can also choose, which changes to commit by supplying the git_log_ids argument. To see all changes to a resource, list them with the git_log_list method.

example

Listing changes to a flow and committing the oldest one with a custom commit message:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    flow = system.flow('my-flow')
    flow.save(
        description='changed description',
    )
    git_log_list = flow.git_log_list(order='timestamp')
    flow.commit(git_log_ids=[git_log_list[0]], message='My commit message')

    return this.success('all done')

Creating and Changing Branches

You can create and change branches in the Engine UI. To do so, open the git action and click on "Manage git branch".

To switch to an already existing branch, select the branch from the drop-down. To create a new branch, type in the branch name and click on "Create"

The resources you see in a git enabled project/bundle/workspace reflect the selected branch. When you switch to a different branch, resources that are not tracked in that branch will be removed. Similarly, resources that are tracked in the new branch but not the old one, will be added.

note

You can also create branches in your local git repository. The Cloudomation UI lists all existing branches in the "Manage git branch" regardless if they were created in the UI or directly in the repository.

Resetting resources in Engine

When you make changes to a resource that turn out to be wrong or unnecessary, you can discard them and restore the resource to the status in git. To do so, go to the resource and hover over the ellipsis next to the commit button. Click on "Reset to status in git".

The extended options of the commit button.

warning

Resetting a resource that was renamed, can overwrite an existing resource if it uses the original name of the renamed resource. E.g. if you rename my-flow to my-flow-2 (without committing), then create a new resource called my-flow, then reset my-flow-2 to the status in git, the content of my-flow will be overwritten.

note

If you want to undo a change that is already committed, you can do so by using regular git commands like git reset or git revert in your local repository and pushing the changes to the remote repository.

Reverting commits and other git actions

If you want to perform more complex actions in git (revert, reset, cherry-pick etc.), you can do so in the local repository. Once you're done, push the changes to the remote repository.

Configuration

Disabling the git integration

To disable the synchronization for a project/bundle/workspace click on "Disable git" from the git actions button.

The button to disable the git integration

danger

If you disable the git integration for a resource, the remote repository will be deleted including all history. This cannot be undone.

Re-synchronising the remote and local repository (needed after renaming the remote repository)

Renaming the remote repository of a synchronised project/bundle/workspace automatically changes the URL of the remote repository. To ensure that you can push changes from your local repository to the remote repository, you need to adjust the remote path of your local repository accordingly.

Example: Changing the remote of your local repository

Let's assume that you have a git enabled project with the remote repository called My project. The corresponding URL is ssh://git@localhost:2022/git/project/My project.git.

You decide to rename the remote repository to My new project. The new URL is now ssh://git@localhost:2022/git/project/My new project.git.

Here's how you can adjust the remote of your local repository to reflect this change:

1. open a command line in the folder where your local repository is
2. (optional) you can display the current remote with the git remote -v command

$ git remote -v
origin	ssh://git@localhost:2022/git/project/My project.git (fetch)
origin	ssh://git@localhost:2022/git/project/My project.git (push)

3. change the remote to the new URL

$ git remote set-url origin 'ssh://git@localhost:2022/git/project/My new project.git'

4. display the remote to confirm that the change is successful

$ git remote -v
origin	ssh://git@localhost:2022/git/project/My new project.git (fetch)
origin	ssh://git@localhost:2022/git/project/My new project.git (push)

Reducing Disk Space Usage

Git is efficient at storing history and in most cases commits don't significantly increase disk usage over time. However, an exceptionally long commit history or commits that include large changes to files can lead to excessive disk usage.

Here are some ways you can reduce the disk usage with native git methods.

danger

The following methods rewrite git history and require you to force push changes into remote repositories. Be sure to understand the implications before proceeding.

Remove Old History

If your git history is very long, you could decide that old commits are not of interest any more. In this case you can choose a cut-off commit and remove any previous commits from the history. This does not affect the current content of your repository.

In your local repository:

1. shallow your repository to the commit of your choice:

   git rev-parse <commit-sha> > .git/shallow

2. remove other branches, tags that reference the deleted commits:

   git gc --prune=now

3. push your changes to the remote:

   git push --force-with-lease
   note

   The option --force-with-lease ensures that you only overwrite the remote if it has not been changed since the last time you fetched it. This is to prevent overwriting commits that you are not aware of.

Remove Specific Files From History

If specific files cause a large portion of the disk usage, you can delete them from the git history. This can be done both with files that still exist or have been deleted already.

note

This method needs a third-party git add-on git-filter-repo.

You can either install it with following their installation guide, or with pip by running

pip install git-filter-repo

In your local repository:

1. (only if the file still exists) rename the file that you want to delete from the history.

   E.g. from my-file.py to my-file.py_save

2. delete the file from the history:

   warning

   If the file still exists, make sure to use the original file name (e.g. my-file.py) in the path argument, otherwise instead of deleteing the file from the history you delete it from the repository.

   git filter-repo --invert-paths --path <path to the original file>

3. (only if the file still exists) rename the file back to its original name

   E.g. from my-file.py_save to my-file.py

4. commit your change locally

5. push your changes to the remote:

   git push --force-with-lease
   note

   The flag --force-with-lease ensures that you only overwrite the remote if it has not changed since the last time you fetched it. This is to prevent overwriting commits of other contributors.

Learn More

Import / Export and Upload

Vault Integration

Email Integration

Plugins

Flows