Git Integration

Using the git integration it is possible to synchronise the content of a git repository with Engine.

Use Cases

Generally it is recommended to store all your Engine resources in git. There are plenty of benefits:

• Use of your favourite offline editor
• Version history of all changes to your automation logic
• Collaboration between users easily possible
• An additional backup of your automation logic
• Branching, merging, rollback, ...

Concept

With git integration configured, all saved changes to resources in Engine are automatically synced to a remote repository that is provided by Engine as part of the workspace. This means, that whenever you change a flow script, enable or disable a schedule, rename a setting etc... a commit is generated and pushed upon saving. The author of the commit is your Engine user.

On the other hand, you can also make changes to resources in your local repository and upon committing and pushing, these changes will be reflected in Engine.

Scope of the synchronisation

You can set up the synchronisation for projects, bundles or the workspace. Only resources within the given project or bundle will be tracked. The git integration for the workspace tracks resources that are not part of a project or a bundle (e.g. users). Activities (such as executions or messages) are not tracked by the git integration.

important

Activities (e.g. executions or messages) are not tracked by the git integration.

Folder structure

Tracked resources can be found in the designated cloudomation-resources folder of the repository. Everything outside of this folder will be ignored by the git integration and can be used for your own purposes e.g. readme files, templates, etc.

note

The resources you will find in cloudomation-resources have the file pattern like any other Cloudomation export i.e. a resource can be represented by of multiple files. For example a flow called my-flow is exported as my-flow.flow-description.txt and my-flow.flow-script.py.

You can read more about Cloudomation export file formats here.

Initial Setup

Per default, the git integration is not set up. To enable the integration navigate to the project/bundle/workspace that you want to synchronize.

Setting up the remote repository

Choose the name of your branch (e.g. develop), write it into the field "Branch name", and save.

The field "Branch name", still empty.

Upon saving a remote repository is created and the Git URL appears below the branch name.

From now on every change you make, that falls into the scope of the synchronisation, will be automatically committed into the remote repository.

Setting up the local repository

To set up the local repository you can clone the remote repository using the "Git URL".

Assigning your public key

Since the cloning uses SSH transfer protocol make sure that your Engine user has your SSH public key assigned.

You can check this by navigating to "Profile" -> "Edit profile". If there are no public keys listed you need to add the content of your public key.

By clicking on "+ Add" you can assign your public key to your profile.

note

You will find your SSH public key under <your home directory>/.ssh/<filename>.pub. If you don't yet have SSH keys, refer to SSH Keygen to generate them.

danger

Never publish your SSH private key. Make sure to assign the public key to your user.

Cloning the remote repository

Once your user has your public SSH key assigned you can create the local repository by cloning the remote.

This can be done on the command line or you can use the Repository Cloning feature of your offline editor (if it has one).

tip

The git URL is automatically put into quotes if the URL has spaces or special characters, so the command line handles it correctly.

Tools like VSCode or TortoiseGit handle quotes differently and might run into an error like fatal: protocol ''ssh' is not supported. In this case remove the quotes from the URL after copying it.

Example: Cloning via command line

1. open a command line in the folder where you want to create the local repository
2. execute the git clone command with the git URL as parameter. E.g.:

git clone "ssh://git@localhost:2022/git/project/Default project.git" Default

Once you have your local repository you can make changes locally, commit and push them to the remote repository. These changes will then be reflected in Engine.

Configuration

Disabling the git integration

To disable the synchronisation for a resource click on "Remove git repository" in the options menu of the resource.

The button to disable the git integration

danger

If you disable the git integration for a resource, the remote repository will be automatically deleted.

Changing branches

If you are working with multiple branches locally you can change which branch gets synced with Engine. The required steps for changing branches in Engine is illustrated in the following example:

Let's assume that you have 2 branches locally prod and develop. In Engine your project is currently synced with prod. Now you want to sync Engine with develop so you can test out new features.

1. First, you need to change the "Branch name" on your project to develop.
2. Fetch & pull changes in your local repository in the branch develop. You'll see a commit on top with the message "Reexport whole content".

note

This commit contains the content of the branch prod since in Engine the resources are still the ones from prod. In the next step we revert this commit so the branch develop is clean again.

3. Run git revert HEAD.
4. Push your local branch develop to the remote repository.

Your project in Engine is now synced with the branch develop.

Re-synchronising the remote and local repository (needed after renaming a project/bundle/workspace)

Renaming a synchronised project/bundle/workspace automatically changes the URL of the remote repository. To ensure that you can push changes from your local repository to the remote repository, you need to adjust the remote path of your local repository accordingly.

Example: Changing the remote of your local repository

Let's assume that you have a git enabled project called My project. The corresponding URL is ssh://git@localhost:2022/git/project/My project.git.

You decide to rename the project to My new project. The URL automatically changes to ssh://git@localhost:2022/git/project/My new project.git.

Here's how you can adjust the remote of your local repository to reflect this change:

1. open a command line in the folder where your local repository is
2. (optional) you can display the current remote with the git remote -v command

$ git remote -v
origin	ssh://git@localhost:2022/git/project/My project.git (fetch)
origin	ssh://git@localhost:2022/git/project/My project.git (push)

3. change the remote to the new URL

$ git remote set-url origin 'ssh://git@localhost:2022/git/project/My new project.git'

4. display the remote to confirm that the change is successful

$ git remote -v
origin	ssh://git@localhost:2022/git/project/My new project.git (fetch)
origin	ssh://git@localhost:2022/git/project/My new project.git (push)

Reducing Disk Space Usage

Git is efficient at storing history and in most cases commits don't significantly increase disk usage over time. However, an exceptionally long commit history or commits that include large changes to files can lead to excessive disk usage.

Here are some ways you can reduce the disk usage with native git methods.

danger

The following methods rewrite git history and require you to force push changes into remote repositories. Be sure to understand the implications before proceeding.

Remove Old History

If your git history is very long, you could decide that old commits are not of interest any more. In this case you can choose a cut-off commit and remove any previous commits from the history. This does not affect the current content of your repository.

In your local repository:

1. shallow your repository to the commit of your choice:

   git rev-parse <commit-sha> > .git/shallow

2. remove other branches, tags that reference the deleted commits:

   git gc --prune=now

3. push your changes to the remote:

   git push --force-with-lease
   note

   The option --force-with-lease ensures that you only overwrite the remote if it has not been changed since the last time you fetched it. This is to prevent overwriting commits that you are not aware of.

Remove Specific Files From History

If specific files cause a large portion of the disk usage, you can delete them from the git history. This can be done both with files that still exist or have been deleted already.

note

This method needs a third-party git add-on git-filter-repo.

You can either install it with following their installation guide, or with pip by running

pip install git-filter-repo

In your local repository:

1. (only if the file still exists) rename the file that you want to delete from the history.

   E.g. from my-file.py to my-file.py_save

2. delete the file from the history:

   warning

   If the file still exists, make sure to use the original file name (e.g. my-file.py) in the path argument, otherwise instead of deleteing the file from the history you delete it from the repository.

   git filter-repo --invert-paths --path <path to the original file>

3. (only if the file still exists) rename the file back to its original name

   E.g. from my-file.py_save to my-file.py

4. commit your change locally

5. push your changes to the remote:

   git push --force-with-lease
   note

   The flag --force-with-lease ensures that you only overwrite the remote if it has not changed since the last time you fetched it. This is to prevent overwriting commits of other contributors.

Learn More

Import / Export and Upload

Vault Integration

Email Integration

Plugins

Flows