Bring WDL apps to the Platform
To get access to this feature, please contact our Support Team.
Overview
Workflow Description Language (WDL) is an open, standardized, human readable and writable language for expressing tasks and workflows. WDL is designed to be a general-purpose workflow language, but it is most widely used in the field of bioinformatics. If you already have tools or execution pipelines that are described in WDL, this page will provide details on how you can push them to the Seven Bridges Platform and run your analysis at scale, using the full potential and benefits of the Seven Bridges execution environment.
Prerequisites
- An account on the Seven Bridges Platform.
- Installed
sbpack
. This is a command-line tool you'll use to adapt and push WDL apps to the Platform. For more details on whatsbpack
can do, how to install it and its main use cases, see About sbpack below. - (Optional) Downloaded WOMtool. The WOMtool
jar
executable is used to generate the JSON input file that specifies all of the WDL app's inputs.
About sbpack
The primary use of sbpack
is to provide an easy way to upload (sbpack
) and download (sbpull
) apps to/from any Seven Bridges powered platform. Since it is a command-line tool, it can be particularly useful as a part of continuous development and integration pipelines for bioinformatics apps, as it allows seamless automated deployment of new app versions to the Platform. It works with apps described using the following workflow description standards:
- Common Workflow Language (CWL). Apart from enabling the standard app pull and push flows, also provides advanced functionalities such as resolution of linked processes, schemadefs and $includes and $imports.
- Nextflow. Adapts, prepares and pushes Nextflow apps for execution in Seven Bridges environments using a special
sbpack_nf
command. - Workflow Description Language (WDL). Uses the
sbpack_wdl
command to convert and push WDL apps for execution in Seven Bridges environments.
To install sbpack
, use the standard install method through pip
:
pip install sbpack
Procedure
The procedure of publishing WDL apps for use on the Platform is a process that consists of the following two stages:
- Initial app conversion. In this step, your WDL app will be converted to a format that is executable on the Platform. However, to optimize your app for execution on the Seven Bridges Platform and make the most out of the Platform's execution environment it is strongly recommended to go through the next step.
- Optimizing the converted app for execution in Seven Bridges environments. The app that has been initially converted now contains an additional configuration file that you will use to define Platform-specific options and fully optimize it for use in the Seven Bridges execution environment. Once the optimized configuration is prepared, the app configuration is pushed to the Platform again.
Initial app conversion
This step adapts the WDL app for execution on the Seven Bridges Platform. It is performed by executing the sbpack_wdl
command in the following format:
sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --womtool-input WOMTOOL_INPUT
In the command above, replace the placeholders as follows:
PROFILE_NAME
is the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file.APPID
specifies the identifier of the app on the Platform, in the{user or division}/{project}/{app_name}
format. If you are using Enterprise, the{user or division}
part is name of your Division on the Platform; otherwise, specify your Platform username. The{project}
part is the project to which you want to push the app and{app_name}
is the name you want to assign to the app. For example the full app ID can bemy-division/my-new-project/my-wdl-app
. If the specified app ID does not exist, it will be created. If it exists, a new revision (version) of the app will be created.WORKFLOW_PATH
needs to be replaced with the path where the WDL app files are located on your local machine. TheWORKFLOW_PATH
folder should include all the dependencies that the workflow needs in order to run.ENTRYPOINT
should be replaced with the path to the actual .wdl file, relative to the root ofWORKFLOW_PATH
.WOMTOOL_INPUT
should be replaced with the path to the JSON inputs file containing your app's inputs schema generated using WOMtool. If you don't have the inputs file, replace--womtool-input
with the--womtool-path
argument and provide the path to the WOMtool jar executable on your local machine, which will generate the inputs and pass them tosbpack_wdl
automatically.
Here is a sample of the command:
sbpack_wdl --profile sbpla --appid sevenbridges-division/wdl-project/test-app --workflow-path /Users/rfranklin/apps/wdl/demo --entrypoint cellranger_workflow.wdl --womtool-input /Users/rfranklin/apps/wdl/demo/inputs.json
Once executed successfully, this command will convert the WDL app for use on the Platform and push it to the Platform project specified as the value of the --appid
argument. The local directory specified as the value of --workflow-path
will now contain an additional sb_wdl_schema.yaml
file. The file contains configuration parameters that can be adjusted and optimized for execution on the Platform.
Optionally, to avoid pushing the app to the Platform at this stage and perform optimizations for the Seven Bridges execution environment beforehand, use the --dump-sb-app
flag at the end of the command. For a full list of available arguments to the sbpack_wdl
command, see the sbpack_wdl command reference.
Optimizing the converted app for execution in Seven Bridges environments
When you have performed the initial conversion step, the generated sb_wdl_schema.yaml
file is important as it contains confguration parameters that will help you optimize the app for execution on the Platform. The file consists of the following major sections:
- The initial section that includes general app information and the documentation content describing the app (if any):
app_content
: Contains details about app's package and WDL file:code_package
: Platform ID of the file that contains the WDL code. This is replaced by thegit_pull
key if the--no-package
option was used to set a git repository as the source of the app's code. See the sbpack_wdl command reference for details.entrypoint
: Relative path to the file containing the WDL code, relative to the root directory of the ZIP file defined incode_package
.class
: Defines the type of workflow description language used for the app. The value will always bewdl
for WDL apps.cwlVersion
: Defines the version of CWL used to describe the app. The value will always beNone
for WDL apps.doc
(Optional): The Markdown-formatted text describing the app.- The
inputs
section that defines details of the app inputs. - The
outputs
section that defines details of app outputs. - The
requirements
section that defines app execution requirements such as initial working directory.
Configuring inputs
Each of the app inputs that is present in the inputs
section contains the following basic details:
id
: Unique identifier of the input.inputBinding
: Defines the mapping of the input to the command line of the app that is being executed. IfinputBinding
is omitted, input is made available in the execution environment, but is not passed on to the WDL executor.prefix
: The ID of the WDL workflow input, in the form that would be added to the WDL inputs JSON file. For example, if you provide a value for an input that is labelled asworkflow.input_file
in the WDL inputs JSON file, the prefix would be defined asworkflow.input_file.
default
: The default value for the input. If the input value is not set on task execution, this default value is taken and passed on to the executor as defined withinputBinding.prefix
.label
: Text description of the input.sbg:toolDefaultValue
: Default value of the input in the WDL workflow. Value provided here is not used in execution and is descriptive (for information purpose) only.sbg:fileTypes
: Comma separated (with spaces) value of file extensions that are used in the file picker when setting up tasks. For example: sbg:fileTypes:“FASTQ, FASTQ.GZ”
.type
: The type of value expected on the input. If there is a question mark at the end, for exampleint?
, the input is optional. Otherwise, the input is mandatory. Learn more about available types of inputs.
To accommodate for the transition between WDL and the Seven Bridges execution environment, the sb_wdl_schema.yaml
file will always contain an additional input whose ID is auxiliary_files
, which contains the list of files not added as explicit workflow inputs but are required for workflow execution. To enable proper execution on the Platform, please do not remove this input from sb_wdl_schema.yaml
.
Note: File and Directory inputs
In some WDL versions, Directory
inputs are marked as File
. The type of such inputs will consequently be File
on the Platform. If the input should actually be a directory, simply change its type
property from File
to Directory
in sb_wdl_schema.yaml
.
There may be cases where input types are not properly set by
sbpack_wdl
during initial conversion, especially for complex input types. These cases may need to be handled manually based on each use case.
Configuring outputs
In addition to executing WDL apps on the Platform, you also need to optimize app outputs to produce and save only files that match the defined criteria, To achieve this and be able to further configure your app outputs, see the details about configuration parameters contained in the outputs
section of the sb_wdl_schema.yaml
file:
id
: Unique identifier of the output. You can change this value to provide a more adequate and descriptive one if necessary.outputBinding
: Defines theglob
expression or pattern that will be used to select the output file.glob
: The glob expression that defines the items to keep as outputs on the output port.
type
: The type of output value.
Example: Configuring an output
The sb_wdl_schema.yaml
file always contains one automatically generated app output:
outputs:
- id: wdl_workdir
outputBinding:
glob: '*'
type: File[]
To configure your outputs, create an entry for each of the output ports you want to have, following the pattern below:
outputs:
- id: csv_file
outputBinding:
glob: '*.csv'
type: File
In the example above, replace csv_file
with an ID that describes your output and replace *.csv
with the glob that matches the type and naming of the data you want to produce on the output. Learn more about available types of outputs.
Example: Configuring a dynamic output directory name
You can also use the sb_wdl_schema.yaml
file to set the name of the output directory by defining it in an app input, provided that the tool itself supports the option of defining the output directory name using the corresponding input argument and its value. The first step is to define the input that takes the output directory name (in the sb_wdl_schema.yaml inputs
section):
inputs:
- id: my_workflow_outdir_name
inputBinding:
prefix: my_workflow.outdir_name
type: string?
Once you have defined the input, define an output, where the output directory glob
will be a variable that gets the value defined in the input above.
outputs:
- id: output_directory
outputBinding:
glob: '$(inputs.outdir_name)'
type: Directory
The $(inputs.outdir)
value is a variable that will be replaced with the actual value entered in the outdir
input when the app is executed.
Configuring requirements
The requirements
sections is primarily used for the following execution-related parameters:
- Setting input staging (making input files available in the app's working directory).
- Setting computation instances that are used for app executions on the Platform.
Setting input staging
Files that are named as inputs to a tool are not, by default, in the tool's working directory. In most apps this access is sufficient, since most tools only need to read their input files, process the data contained in them, and write new output files on the basis of this data to their working directory. However, in some cases an app might require input files to be placed directly in its working directory. If this is the case with your app, modify the requirements
section in the sb_wdl_schema.yaml
file as follows:
requirements:
- class: InitialWorkDirRequirement
listing:
- '$(inputs.tsv_files)'
Entries under listing
define files and directories that will be made available in the app’s working directory before the command is executed. The files and directories are usually defined as variables named after their respective input IDs.
Another useful option is creation of a file directly in the working directory. This is done by defining entryname
and entry
keys in the InitialWorkDirRequirement
class, as follows:
requirements:
- class: InitialWorkDirRequirement
listing:
- entryname: input_nanoseq.csv
entry: |
${
if (inputs.auxiliary_files && !inputs.in_csv_file){
var content = 'group,barcode';
for (var i = 0; i < inputs.auxiliary_files.length; i++){
if (inputs.auxiliary_files[i].metadata['barcode']){
var barcode = inputs.auxiliary_files[i].metadata['barcode'];
}
else {
var barcode = '';
}
if (inputs.auxiliary_files[i].metadata['group']){
var group = inputs.auxiliary_files[i].metadata['group'];
}
else {
var group = '';
}
content = content.concat(group,',',barcode,'\\n');
}
return content
}
else {
return ''
}
In the example code above, entryname
defines the name of the file generated in the working directory, which is input_nanoseq.csv
, while entry
contains a Javascript expression that populates the generated file by getting barcode
and group
metadata values from input files and concatenating them in a single CSV file. As we're using a Javascript expression, it is recommended to use the default option of generating a YAML schema when using the sbpack_wdl
command, as YAML is more convenient and reduces the possibility of making errors compared to using Javascript expressions in a JSON file. The expression can be defined to match your needs and intended use. Read more about dynamic expressions in tool descriptions or see some of the most common expression examples in our Javascript Cookbook.
Setting execution instances
Another useful option that is available for configuration in the hints
section is the definition of the computation instance used for app execution on the Platform. This is also done by defining key-value pairs as follows:
hints:
- class: sbg:AWSInstanceType
value: c4.8xlarge;ebs-gp2;2000
In this case, the workflow uses a c4.8xlarge
instance with 2000 GB of attached EBS storage. The value consists of the following three parts (separated by ;
):
- Instance type, e.g.
c4.8xlarge
. - Attached disk type: always
ebs-gp2
for all instances with EBS storage. - Disk size in GB.
See the list of AWS US and AWS EU (Available on Seven Bridges Platform EU) instances that are available for task execution on the Platform.
Pushing the optimized app configuration to the Platform
When you are done with changes to the sb_wdl_schema.yaml
file, push the optimized app configuration to the Platform. As we just making configuration changes to an app that has already been pushed to the Platform, this can be done using the regular sbpack
command in the following format:
sbpack PROFILE_NAME APPID CONFIG_FILE
In the command above, PROFILE_NAME
refers to the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file. The APPID
parameter specifies the identifier of the app on the Platform. Use the same APPID
you used in the initial conversion step. Finally, CONFIG_FILE
is the sb_wdl_schema.yaml
in which you made app execution optimizations. The final command should be, for example:
sbpack sbpla-profile sevenbridges-division/wdl-project/test-app sb_wdl_schema.yaml
This pushes the modified app configuration to the Platform and creates a new revision (version) of the app. Once this is done, you are ready to run a task using the app.
Updating already converted and optimized apps
If you have already converted your app, made optimizations in the sb_wdl_schema.yaml
file, and pushed the app to the Platform, all subsequent updates to the app's WDL code and the process of propagating the update to the Platform are quite straightforward. If the updates you made do not require changes to manually configured parameters in the sb_wdl_schema.yaml
file (such as inputs, outputs, requirements, etc.), create a new code package by running a command in the following format:
sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --sb-schema SB_SCHEMA
This command is almost the same as the initial app conversion step, but differs in the additional --sb-schema
argument. This argument allows you to provide and reuse an existing sb_wdl_schema.yaml
configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on the Seven Bridges Platform. The command will generate a new code package based on your updated WDL code provided through --workflow-path
and --entrypoint
and the YAML or JSON configuration file provided through --sb-schema
, and push the updated app to the Platform creating a new revision (version).
Copying WDL apps between projects on the Platform
When an app is on the Platform, you can copy it and use it on other Platform projects. To copy WDL apps between projects, use the sbcopy
command that is a part of the sbpack
utility:
sbcopy [--profile PROFILE] --appid APPID --projectid PROJECTID
The command takes the following arguments:
PROFILE
: refers to the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file.APPID
: specifies the identifier of the app on the Platform. Takes the form{user or division}/{project}/{app_id}
. If you are using Enterprise, the{user or division}
part is name of your Division on the Platform; otherwise, specify your Platform username. The{project}
part is the source project where the app is located and{app_id}
is the ID of the app you want to copy; for examplemy-division/my-new-project/my-wdl-app
.PROJECTID
: is the identifier of the destination project where the app will be copied. Takes the form{user or division}/{project}
. If you are using Enterprise, the{user or division}
part is name of your Division on the Platform; otherwise, specify your Platform username.
The final command should be, for example:
sbcopy sbpla-profile sevenbridges-division/wdl-project/test-app other-division/my-wdl-project
Note that WDL app copies made through standard visual interface or API methods instead of using
sbcopy
will still point to the originally pushed code package and the original project where it is located. This might cause failures due to lack of permissions, if users who need to run the copied instances of the app aren't added to the project where the original code package is located. To avoid this, please usesbcopy
to copy WDL apps between projects on the Platform, as described above.
sbpack_wdl
command reference
sbpack_wdl
command referenceHere is a list describing all available arguments od the sbpack_wdl
command that is used to convert and push WDL apps for execution on the Seven Bridges Platform.
Argument | Required | Description |
---|---|---|
-h , --help | Shows the list of all arguments and their corresponding explanations. | |
--profile PROFILE | Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file. If you are using the default profile, this parameter can be omitted. | |
--appid APPID | Required | The ID of the WDL app once it is pushed to the Seven Bridges Platform. Takes the form {user or division}/{project}/{app_id} . If you are using Enterprise, the {user or division} part is name of your Division on the Platform; otherwise, specify your Platform username. The {project} part is the project to which you want to push the app and {app_id} is the ID you want to assign to the app, for example my-division/my-new-project/my-wdl-app . |
--workflow-path WORKFLOW_PATH | Required | Path to the main workflow directory (the local directory where the app's files are located). |
--entrypoint ENTRYPOINT | Required | Relative path to the the file that contains the app's WDL code from the main workflow directory defined in --workflow-path . |
--sb-package-id SB_PACKAGE_ID | ID of an already uploaded package. If you have already converted and pushed the app to the Platform, it has its own code package ID, as shown in the code_package key in the sb_wdl_schema.yaml file. When the package ID is provided, the conversion script will skip the upload step and thus take less time to execute. | |
--sb-doc SB_DOC | Path to the app description document written in Markdown. The document is meant to provide additional details about the app and will be shown when viewing app details on the Platform. If not provided, README.md will be used if available in the same directory where entrypoint file is located. | |
--womtool-input WOMTOOL_INPUT | Required if --womtool-path isn't used | Path to JSON inputs file containing your app's inputs schema generated using WOMtool. |
--womtool-path WOMTOOL_PATH | Required if --womtool-input isn't used | Path to the WOMtool jar file on your local machine. If you don't already have a JSON inputs file that you can provide via --womtool-input , this command will use WOMtool to generate the file on the fly and provide it to sbpack_wdl automatically. |
--sb-schema SB_SCHEMA | Path to an existing sb_wdl_schema file in JSON or YAML format. This allows you to use an existing configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on the Seven Bridges Platform. | |
--dump-sb-app | Dumps the converted app to a local file without pushing it to the Platform. Using this option will enable you to convert the app and generate the sb_wdl_schema.yaml file to make configuration optimizations prior to pushing the app the Platform. | |
--no-package | References a git repository containing the app's code instead of pushing the app's code to the Platform. Git repository address is specified as the git_pull key in the sb_wdl_schema.yaml file, instead of code_package . For example: git_pull: https://git.domain.com/repository . The value for the git_pull key is the URL you would normally use to clone the repository to your local environment. | |
--json | Creates the sb_wdl_schema file in JSON format instead of the default YAML. |
Important notes for executing WDL apps on the Seven Bridges Platform
- Workflows are executed in Local mode. Make sure your workflow can run in Local mode before porting it to the Seven Bridges Platform.
- Use of Docker is required. See how to create and upload a Docker image containing your app and make sure to edit the WDL code to use the newly created image. If the Docker image is not specified for a process, a default alpine image will be used.
- Execution is done in a dedicated working directory and all the WDL work is done in the
wdl_workdir
directory inside the working directory. Avoid using and relying on hard-coded paths in workflows.
Differences between running CWL and WDL apps on the Platform
Executions on the Platform normally result in separate jobs (steps) being created for each tool in the workflow. When a WDL pipeline is executed on the Platform, a single execution job will be created, regardless of the number of tools within the pipeline. This "one app, one job" approach results in the following differences compared to CWL app executions:
- Memoization is not available for WDL apps. As memoization relies on using previous job outputs to skip identical jobs in new executions, it is not useful in situations when there is only one job in an app execution.
- Handling of spot instance interruption can't be used with WDL apps. If a spot instance gets terminated when a WDL app is running, the app would have to be rerun on an on-demand instance from the beginning, which would result in increased costs instead of savings. Therefore, it is recommended to disable spot instances for WDL app executions.
- Task stats and logs are organized differently. As task statistics and logs are usually shown and organized per job, WDL apps will have cumulative stats for the single job in the execution, while logs will be saved in a single folder.
Updated over 2 years ago