Scheduling

Engine supports flexible and powerful means of repeatedly running a flow. If you want to set up a schedule without reading through the underlying concept you can refer to the scheduling bundle. The included scheduler comes with a form for configuring the schedule in a user-friendly way.

To better understand the full capabilities of scheduling with Engine, simply read on.

Use Cases

Use a Schedule to

• Execute a flow in fixed or flexible time intervals (minutes, hours, days, weeks, months, etc...),
• Execute a flow on particular days of a calendar,
• Execute a flow in arbitrary complex intervals,
• Permanently poll a third party system.

Concept

Engine separates

• the logic which ensures a flow is executed repeatedly (the Scheduler)
• the logic of the flow itself (the Flow)
• the resource which the Flow and the Scheduler, and stores the configuration values (the Schedule)

Using this separation it is possible to use the same scheduler logic (e.g. schedule something daily) for different flows with varying settings (e.g. scheduled at different hours)

note

As soon as a schedule is enabled Engine will make sure that an active execution of the scheduler exists. If the scheduled execution fails or is cancelled, Engine will re-start it for you.

Scheduler

A scheduler contains a script which implements logic to wait until conditions are met (e.g. a certain time has passed) and then start the flow.

When a scheduler is started, it receives the IDs of the referenced flow and schedule, as well as the configuration values from the schedule.

With this information, the scheduler can implement arbitrary logic to wait until the conditions are met.

note

Cancelling a scheduler execution will result in Engine starting another scheduler execution after one minute. To stop the schedule, the schedule resource must be disabled or deleted.

example

Here is an example of a simple scheduler script which starts the flow every minute. Note that once the scheduler execution is finished (after the scheduled flow has finished), the scheduler execution will be restarted by Engine.

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
    # The execution receives the IDs of the referenced flow and schedule.
    # In this example we'll only use the flow_id
    flow = system.flow(inputs['flow_id'], by='id')

    # wait one minute
    this.sleep(60)

    # start the flow
    this.flow(flow.get('name'))

    return this.success('all done')

Flow

The flow contains the script which is repeatedly executed.

A scheduler might pass context information to the flow execution. Which inputs are passed (if any) is up to the Scheduler script. The information passed can be used to have different behaviour in different iterations or for logging/audit purposes.

Here are some examples:

• The iteration counter
• A flag if the current day is a public holiday
• Email of the operator on shift

Schedule

The schedule links one scheduler with and one flow. It also stores the configuration values for the scheduler.

Configuration Values

Different schedulers can support different options. E.g. the number of minutes between two iterations, or the time at which to start the flow every day.

example

time: '09:00'
timezone: 'Europe/Vienna'
days_of_week: [1, 2, 3, 4, 5]

Note that the configuration values can be empty if the scheduler does not have any configurable parameters (i.e. the whole scheduling logic is described within the scheduler). However, it is recommended that parameters are not hard-coded within schedulers but are instead parameterizable via the configuration values for flexibility and maintainability.

Enabling and Disabling Schedules

The schedule can be disabled or enabled. Once the schedule is enabled and saved, Engine will immediately start an execution of the scheduler script.

When a schedule is disabled, the currently running execution of the scheduler script iscancelled by Engine.

note

You cannot change fields of a schedule while it is running. To change the configuration or use a different scheduler or flow, you must first disable the schedule.

Next Runs

Schedules have a dedicated field where the timestamps of the next runs can be saved. The logic for storing the next run time must be implemented within the scheduler. In the scheduling-bundle-scheduler (part of the Scheduling bundle) the logic is implemented to store the next 10 runs.

Example: a Simple Recurring Scheduler

Let's create a simple scheduler that takes one configuration value: the number of minutes between two runs.

First, create a scheduler resource in the UI.

Next, let's define the scheduler script:

import flow_api

def handler(system: flow_api.System, this: flow_api.Execution, inputs: dict):
  this.flow(
    inputs['flow_id'],
    flow_by='id',
  )
  interval_minutes = inputs['configuration']['interval_minutes'] # get the interval from the configuration
  this.sleep(interval_minutes*60)

When the scheduler is executed by the schedule, it will get the configuration values from the schedule. The script expects an interval_minutes configuration value.

To make the configuration easy, let's define an input schema for the scheduler. This way the user can adjust the configuration of the schedule in the UI.

element: form-object
required:
  - interval_minutes
properties:
  interval_minutes:
    label: Interval minutes
    element: form-integer
    type: integer
    default: 30
    description: How many minutes to wait between executions.

Now you can create a schedule that uses this scheduler and easily configure the interval minutes in the UI.

using the scheduler in a schedule

Complex Schedulers

A scheduler script uses the same syntax as a regular flow script. This means that a scheduler can use arbitrary complex logic, connect to third party systems, and start child executions. Some examples:

• Connect to a calendar to check the next appointment time
• Check a weather API when some flow should only run after rain
• Analyse the status of other executions
• Check an email inbox

There are two recommendations to keep in mind when designing a scheduler:

1. Separation of concerns

The scheduler should only contain logic needed to do the scheduling. The referenced flow on the other hand contains the logic which is scheduled.

2. Reusability

A scheduler should be designed so it can be used for different flows.

Considering those points the examples above should rather be placed in the Flow which is scheduled, than in the scheduler.

Scheduling Bundle

We provide a comprehensive scheduler the scheduling-bundle-scheduler as part of the freely downloadable Scheduling bundle.

This scheduler gives you a lot of flexibility and ease of use. Some example parameters are:

• time of the day,
• calender weeks,
• holiday calenders,
• and many more.

With that you can easily configure a schedule that e.g. runs a flow at noon every weekday that is not a public holiday and falls in Q2 of the year.

note

The scheduler included in the scheduling bundle is implemented in a way that the scheduler execution does not end when the scheduled flow has finished:

'''
approximate logic of scheduling-bundle-scheduler
'''
while True:
  # calculate and sleep until next run
  this.flow(flow_name)

This can lead to the scheduler running for a long time. To avoid the scheduler execution growing indefinitely (resulting from logs and savepoints), the scheduler supports a configurable maximum runtime restart_scheduler_execution_days that can be set in the setting. The default is 28 days, after which the scheduler will restart itself.

Setting up a New Schedule

To set up a new schedule navigate to the flow that you want scheduled and start the process with the help of the plugin action "Create Schedule".

the plugin action in the flow

A new schedule window opens with the flow already selected. If the scheduling bundle is installed, the schedule-bundle-scheduler is also selected as the scheduler. You can configure the schedule easily with the help of the form.

the form for configuring the schedule

Once you have configured the schedule, enable it and save it. Engine will immediately start an execution of the scheduler script.

Next Runs

Once the schedule is enabled, the next runs field of the schedule will be updated with the timestamps of the next runs.

the next runs field

Schedule Overview via Advanced Search

If your usecase invloves scheduling many tasks, you can use the advanced search to get an overview of the next runs of your schedules. You can also sort and filter them based on criteria like the next run time, the schedule name, the flow name, and much more.

note

If you cannot see the "Scheduled for" column in your execution live monitor, show it by clicking on the following icon:

Here are some examples:

Upcoming Schedules

Filters:

• Status: NOT ENDED (unselect ended executions)
• Execution Type: SCHEDULE (select only schedule)
• (optional) Location: <project_name>

Sort by:

• Scheduled for column (click on the column header until it is sorted in ascending order)

overview of upcoming schedules

Upcoming Schedules on Specific Day

Use the same filters as above but add "After" and "Before" filters with the specific date.

overview of upcoming schedules on a specific day

Learn More

Executions

Webhooks

Scheduling bundle