About task execution

Upon execution, your task gets broken down into modular units called jobs. Each job is then executed when its inputs become available. Jobs are carried out on compute instances, a process orchestrated by the Scheduling Algorithm. You can control task execution by setting certain parameters that the Scheduling Algorithm bases its decisions on.

Jobs

Jobs are the various steps of your task execution and include tool executions, file uploading/downloading, and retrieval of Docker images.

A job is created for every tool execution. Most of the time a single tool will execute only once per workflow, thus giving a single job per task.

In some cases, however, a tool may be run multiple times to process individual parts of a large input more efficiently. This yields multiple jobs per tool. A tool may also be executed multiple times if one of its inputs is a list whose elements are to be processed in parallel.

Instances

When you run an analysis, it gets executed on a computational instance from the cloud infrastructure provider (Amazon Web Services).

The computation instances appear as remote computers capable of executing generic software and are referred to as instances.

As with any computer, an instance will have a number of CPU cores, memory, hard disk, and network resources.

Seven Bridges provides the infrastructure that controls instances throughout the execution of your analysis.

Queueing

There are several cases in which a task can be temporarily queued:

1. The task has been just submitted and is awaiting execution.
2. The maximum number of parallel instances for your account has been reached.
3. Some of the cloud infrastructure resources required for task execution are not available.

In cases 2 and 3 above, task status will change back from QUEUED to RUNNING when the required parallel instances or cloud infrastructure resources become available. This change of task status can happen several times during execution.

Scheduling

The process of assigning and launching instances that fit the tool executions in a task is called scheduling.

The orchestration of job execution as well as instance provision is carried out by the Seven Bridges bespoke Scheduling Algorithm.

When a tool is about to be executed, the Scheduling Algorithm picks the best instance based on the tool's resource requirements. The Scheduling Algorithm may launch that job on an active instance, side-by-side with an earlier job from the same task, or it may decide to launch a new instance to host the new job.

Controlling execution

You can control the execution of your task by customizing parameters the Seven Bridges Scheduling Algorithm works with.

The parameters you can tune are the types of instance(s) used for your analysis, as well as how many of each instance type can be used in parallel.

These parameters can be attached to tools and workflows that the Scheduling Algorithm will respect when deciding which tool runs where. Those parameters are called execution hints.

The Seven Bridges Platform understands and implements tools, workflows, jobs and execution as prescribed by the Common Workflow Language (CWL).