Set execution mode for Nextflow apps

Overview

Nextflow implementation on the Seven Bridges Platform allows you to choose between single-instance or multi-instance task execution. The multi-instance mode will shorten the total task running time by using multiple computation instances at the same time to distribute the workload and produce results more quickly and efficiently.

Nextflow execution mode can be set on two levels:

• As an app-level setting that becomes the default execution mode setting for each new task that involves the app.
• Individually in task Execution Settings for each new task run. Note that execution mode set at task level overrides the execution mode configured at app level. This means that an app that has multi-instance execution mode set as the default value at app level will still be executed in single-instance mode if single-instance is set as the execution mode value for a specific task (execution) of the app.

If Nextflow execution mode isn't explicitly set at either of the two levels, multi-instance mode is used as the default.

Set execution mode at app level

Nextflow execution mode can be set at the app level, meaning that the app will run in the defined execution mode by default, unless specified otherwise at task level, when executing the app. There are two ways to define the execution mode for an app, when performing initial app conversion to Nextflow using the sbpack tool, or during the optimization step that takes place after the initial conversion.

Set execution mode during initial app conversion using sbpack

To set the app execution mode of a Nextflow app at this stage of bringing it to the Platform, follow these steps:

1. When installing sbpack, make sure to install the latest version:
   Shell

   pip install --upgrade sbpack
   

2. When performing the initial app conversion, add the --execution-mode argument with the value set to multi-instance, for example:
   Shell

   sbpack_nf --appid sevenbridges-division/nextflow-project/test-app --workflow-path /Users/rfranklin/apps/nextflow/demo --entrypoint app.nf --execution-mode multi-instance
   

This sets multiple instances as the default execution mode for the app. To configure single-instance execution as the default app setting, set the value to single-instance instead.

Set execution mode during the app optimization step

1. During the optimization step, add the following hint to sb_nextflow_schema.yaml:
   YAML

   hints:
     - class: sbg:NextflowExecutionMode
       value: multi-instance
   

2. Push the configured app to the Platform. This sets multiple instances as the default execution mode. To configure single-instance execution as the default app setting, add the following hint to sb_nextflow_schema.yaml:
   YAML

   hints:
     - class: sbg:NextflowExecutionMode
       value: single-instance
   

Set execution mode at task level

Through the visual interface

Single-instance or multi-instance mode can be set at task level for each individual execution of a Nextflow app. If execution mode is defined at the app level as well, tesk-level setting has higher priority and overrides the app-level setting.

To configure Nextflow execution mode for a task, follow these steps:

1. Find the Nextflow app you want to use in your analysis.
2. Click Run. This creates a draft task.
3. Open the Execution Settings tab.
4. In the Nextflow Multi-Instance Execution section, set the value to On. This enables multi-instance execution for the task.
   

Through the API

Nextflow execution mode can also be specified when creating a new task through the API, using the nextflow_execution_mode parameter in the tasks execution settings. Here are the details of the API call:

Request body

To set Nextflow execution mode and run the task through the API, use the standard Create a new task API call, with the addition of the nextflow_execution_mode parameter in the execution_settings object. The nextflow_execution_mode parameter can have the following two values:

• SINGLE_INSTANCE
• MULTI_INSTANCE

Here is an example request body with Nextflow execution set to multi-instance mode:

{   
	"description": "My Nextflow task",
	"name": "Nextflow run Oct 31",
	"app": "rfranklin/my-project/nextflow-app",
	"project": "rfranklin/my-project",
	"use_interruptible_instances": false,
	"execution_settings": {
		"nextflow_execution_mode": "MULTI_INSTANCE",
		"instance_type": "c4.2xlarge;ebs-gp2;2000",
		"max_parallel_instances": 1
	},
	"inputs": {
		"input_zip": {
			"class": "File",
			"path": "567890abc3d8130ea4047731",
			"name": "example_input_file.zip"
		}
	}
}