Bring WDL apps to the Platform

πŸ“˜

To get access to this feature, please contact our Support Team.

Overview

Workflow Description Language (WDL) is an open, standardized,Β human readable and writableΒ language for expressing tasks and workflows. WDL is designed to be a general-purpose workflow language, but it is most widely used in the field of bioinformatics. If you already have tools or execution pipelines that are described in WDL, this page will provide details on how you can push them to the Seven Bridges Platform and run your analysis at scale, using the full potential and benefits of the Seven Bridges execution environment.

Prerequisites

  • An account on the Seven Bridges Platform.
  • Installed sbpack. This is a command-line tool you'll use to adapt and push WDL apps to the Platform. For more details on what sbpack can do, how to install it and its main use cases, see About sbpack below.
  • (Optional) Downloaded WOMtool. The WOMtoolΒ jar executable is used to generate the JSON input file that specifies all of the WDL app's inputs.

About sbpack

The primary use of sbpack is to provide an easy way to upload (sbpack) and download (sbpull) apps to/from any Seven Bridges powered platform. Since it is a command-line tool, it can be particularly useful as a part of continuous development and integration pipelines for bioinformatics apps, as it allows seamless automated deployment of new app versions to the Platform. It works with apps described using the following workflow description standards:

  • Common Workflow Language (CWL). Apart from enabling the standard app pull and push flows, also provides advanced functionalities such as resolution of linked processes, schemadefs and $includes and $imports.
  • Nextflow. Adapts, prepares and pushes Nextflow apps for execution in Seven Bridges environments using a special sbpack_nf command.
  • Workflow Description Language (WDL). Uses theΒ sbpack_wdl command to convert and push WDL apps forΒ execution in Seven Bridges environments.

To install sbpack, use the standard install method through pip:

pip install sbpack

Procedure

The procedure of publishing WDL apps for use on the Platform is a process that consists of the following two stages:

  • Initial app conversion. In this step, your WDL app will be converted to a format that is executable on the Platform. However, to optimize your app for execution on the Seven Bridges Platform and make the most out of the Platform's execution environment it isΒ strongly recommended to go through the next step.
  • Optimizing the converted app for execution in Seven Bridges environments. The app that has been initially converted now contains an additional configuration file that you will use to define Platform-specific options and fully optimize it for use in the Seven Bridges execution environment. Once the optimized configuration is prepared, the app configuration is pushed to the Platform again.

Initial app conversion

This step adapts the WDL app for execution on the Seven Bridges Platform. It is performed by executing theΒ sbpack_wdlΒ command in the following format:

sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --womtool-input WOMTOOL_INPUT

In the command above, replace the placeholders as follows:

  • PROFILE_NAME is the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file.
  • APPIDΒ specifies the identifier of the app on the Platform, in theΒ {user or division}/{project}/{app_name}Β format.Β If you are using Enterprise, theΒ {user or division}Β part is name of your Division on the Platform; otherwise, specify your Platform username. TheΒ {project}Β part is the project to which you want to push the app andΒ {app_name}Β is the name you want to assign to the app. For example the full app ID can beΒ my-division/my-new-project/my-wdl-app. If the specified app ID does not exist, it will be created. If it exists, a newΒ revision (version)Β of the app will be created.
  • WORKFLOW_PATHΒ needs to be replaced with the path where the WDL app files are located on your local machine.Β The WORKFLOW_PATHΒ folder should include all the dependencies that the workflow needs in order to run.
  • ENTRYPOINTΒ should be replaced with the path to the actual .wdl file, relative to the root ofΒ WORKFLOW_PATH.
  • WOMTOOL_INPUTΒ should be replaced with the path toΒ the JSON inputs file containing your app's inputs schema generated using WOMtool. If you don't have the inputs file, replaceΒ --womtool-inputΒ with theΒ --womtool-pathΒ argument and provide the path to theΒ WOMtoolΒ jar executable on your local machine, which will generate the inputs and pass them toΒ sbpack_wdl automatically.

Here is a sample of the command:

sbpack_wdl --profile sbpla --appid sevenbridges-division/wdl-project/test-app --workflow-path /Users/rfranklin/apps/wdl/demo --entrypoint cellranger_workflow.wdl --womtool-input /Users/rfranklin/apps/wdl/demo/inputs.json

Once executed successfully, this command will convert the WDL app for use on the Platform and push it to the Platform project specified as the value of the --appid argument. The local directory specified as the value ofΒ --workflow-path will now contain an additional sb_wdl_schema.yamlΒ file. TheΒ file contains configuration parameters that can be adjusted and optimized for execution on the Platform.

Optionally, to avoid pushing the app to the Platform at this stage and perform optimizations for the Seven Bridges execution environment beforehand, use theΒ --dump-sb-app flag at the end of the command. For a full list of available arguments to the sbpack_wdlΒ command, see the sbpack_wdl command reference.

Optimizing the converted app for execution in Seven Bridges environments

When you have performed the initial conversion step, the generatedΒ sb_wdl_schema.yamlΒ file is important as it contains confguration parameters that will help you optimize the app for execution on the Platform. The file consists of the following major sections:

  • The initial section that includes general app information and the documentation content describing the app (if any):
    • app_content: Contains details about app's package and WDL file:Β Β 
    • code_package: Platform ID of the file that contains the WDL code. This is replaced by theΒ git_pull key if theΒ --no-package option was used to set a git repository as the source of the app's code. See the sbpack_wdl command reference for details.
    • entrypoint:Β Relative path to the file containing the WDL code, relative to the root directory of the ZIP file defined in code_package.
    • class: Defines the type of workflow description language used for the app. The value will always beΒ wdlΒ for WDL apps.
    • cwlVersion: Defines the version of CWL used to describe the app.Β The value will always beΒ NoneΒ for WDL apps.
    • docΒ (Optional): The Markdown-formatted text describing the app.Β 
    • TheΒ inputs section that defines details of the app inputs.
    • TheΒ outputs section that defines details of app outputs.
    • TheΒ requirements section that defines app execution requirements such as initial working directory.

Configuring inputs

Each of the app inputs that is present in theΒ inputs section contains the following basic details:

  • id: Unique identifier of the input.
  • inputBinding: Defines the mapping of the input to the command line of the app that is being executed.Β If inputBindingΒ is omitted, input is made available in the execution environment, but is not passed on to the WDL executor.
    • prefix:Β The ID of the WDL workflow input, in the form that would be added to the WDL inputs JSON file. For example, if you provide a value for an input that is labelled as workflow.input_fileΒ in the WDL inputs JSON file,Β the prefix would be defined as workflow.input_file.
  • default: The default value for the input. If the input value is not set on task execution, this default value is taken and passed on to the executor as defined withΒ inputBinding.prefix.
  • label: Text description of the input.Β 
  • sbg:toolDefaultValue:Β Default value of the input in the WDL workflow. Value provided here is not used in execution and is descriptive (for information purpose) only.
  • sbg:fileTypes: Comma separated (with spaces) value of file extensions that are used in the file picker when setting up tasks. For example: sbg:fileTypes: β€œFASTQ, FASTQ.GZ”.
  • type: The type of value expected on the input. If there is a question mark at the end, for exampleΒ int?, the input is optional. Otherwise, the input is mandatory. Learn more about available types of inputs.

To accommodate for the transition between WDL and the Seven Bridges execution environment,Β theΒ sb_wdl_schema.yamlΒ file will always contain an additional input whose ID isΒ auxiliary_files, which contains the list of files not added as explicit workflow inputs but are required for workflow execution. To enable proper execution on the Platform, please do not remove this input from sb_wdl_schema.yaml.

Note: File and Directory inputs

In some WDL versions, DirectoryΒ inputs are marked as File. The type of such inputs will consequently beΒ File on the Platform. If the input should actually be a directory, simply change itsΒ type property fromΒ File toΒ Directory inΒ sb_wdl_schema.yaml.

πŸ“˜

There may be cases where input types are not properly set by sbpack_wdlΒ during initial conversion, especially for complex input types. These cases may need to be handled manually based on each use case.

Configuring outputs

In addition to executing WDL apps on the Platform, you also need to optimize app outputs to produce and save only files that match the defined criteria, To achieve this and be able to further configure your app outputs, see the details about configuration parameters contained in theΒ outputs section of theΒ sb_wdl_schema.yamlΒ file:

  • id: Unique identifier of the output. You can change this value to provide a more adequate and descriptive one if necessary.
  • outputBinding: Defines the globΒ expression or pattern that will be used to select the output file.
    • glob: The glob expression that defines the items to keep as outputs on the output port.
  • type: The type of output value.

Example: Configuring an output

TheΒ sb_wdl_schema.yamlΒ file always contains one automatically generated app output:

outputs:
  - id: wdl_workdir
    outputBinding:
        glob: '*'
    type: File[]

To configure your outputs, create an entry for each of the output ports you want to have, following the pattern below:

outputs:
  - id: csv_file
    outputBinding:
        glob: '*.csv'
    type: File

In the example above, replace csv_fileΒ with an ID that describes your output and replaceΒ *.csvΒ with the glob that matches the type and naming of the data you want to produce on the output. Learn more about available types of outputs.

Example: Configuring a dynamic output directory name

You can also use theΒ sb_wdl_schema.yamlΒ file to set the name of the output directory by defining it in an app input, provided that the tool itself supports the option of defining the output directory name using the corresponding input argument and its value. The first step is to define the input that takes the output directory name (in theΒ sb_wdl_schema.yaml inputs section):

inputs:
  - id: my_workflow_outdir_name
    inputBinding:
        prefix: my_workflow.outdir_name
    type: string?

Once you have defined the input, define an output, where the output directory globΒ will be a variable that gets the value defined in the input above.

outputs:
  - id: output_directory
    outputBinding:
        glob: '$(inputs.outdir_name)'
    type: Directory

TheΒ $(inputs.outdir) value is a variable that will be replaced with the actual value entered in theΒ outdir input when the app is executed.

Configuring requirements

TheΒ requirements sections is primarily used for the following execution-related parameters:

  • Setting input staging (making input files available in the app's working directory).
  • Setting computation instances that are used for app executions on the Platform.

Setting input staging

Files that are named as inputs to a tool are not, by default, in the tool's working directory. In most apps this access is sufficient, since most tools only need to read their input files, process the data contained in them, and write new output files on the basis of this data to their working directory. However, in some cases an app might require input files to be placed directly in its working directory. If this is the case with your app, modify theΒ requirements section in theΒ sb_wdl_schema.yamlΒ file as follows:

requirements:
  - class: InitialWorkDirRequirement
    listing:
      - '$(inputs.tsv_files)'

Entries under listingΒ define files and directories that will be made available in the app’s working directory before the command is executed. The files and directories are usually defined as variables named after their respective input IDs.

Another useful option is creation of a file directly in the working directory. This is done by defining entryname andΒ entry keys in the InitialWorkDirRequirementΒ class, as follows:

requirements:
- class: InitialWorkDirRequirement
  listing:
  - entryname: input_nanoseq.csv   
    entry: |
    ${
    if (inputs.auxiliary_files && !inputs.in_csv_file){
        var content = 'group,barcode';
        for (var i = 0; i  < inputs.auxiliary_files.length; i++){
            if (inputs.auxiliary_files[i].metadata['barcode']){
                var barcode = inputs.auxiliary_files[i].metadata['barcode'];
            }
            else {
                var barcode = '';
            }
            if (inputs.auxiliary_files[i].metadata['group']){
                var group = inputs.auxiliary_files[i].metadata['group'];
            }
            else {
                var group = '';
            }
            content = content.concat(group,',',barcode,'\\n');
        }
        return content
    }
    else {
        return ''
    }

In the example code above, entrynameΒ defines the name of the file generated in the working directory, which isΒ input_nanoseq.csv, whileΒ entry contains a Javascript expression that populates the generated file by getting barcodeΒ and groupΒ metadata values from input files and concatenating them in a single CSV file. As we're using a Javascript expression, it is recommended to use the default option of generating a YAML schema when using the sbpack_wdl command, as YAML is more convenient and reduces the possibility of making errors compared to using Javascript expressions in a JSON file. The expression can be defined to match your needs and intended use. Read more about dynamic expressions in tool descriptions or see some of the most common expression examples in our Javascript Cookbook.

Setting execution instances

Another useful option that is available for configuration in theΒ hints section is the definition of the computation instance used for app execution on the Platform. This is also done by defining key-value pairs as follows:

hints:
  - class: sbg:AWSInstanceType
    value: c4.8xlarge;ebs-gp2;2000

In this case, the workflow uses a c4.8xlargeΒ instance with 2000 GB of attached EBS storage. The value consists of the following three parts (separated by ;):

  • Instance type, e.g. c4.8xlarge.
  • Attached disk type: alwaysΒ ebs-gp2 for all instances with EBS storage.
  • Disk size in GB.

See the list ofΒ AWS US and AWS EU (Available on Seven Bridges Platform EU) instances that are available for task execution on the Platform.Β 

Pushing the optimized app configuration to the Platform

When you are done with changes to theΒ sb_wdl_schema.yamlΒ file, push the optimized app configuration to the Platform. As we just making configuration changes to an app that has already been pushed to the Platform, this can be done using the regularΒ sbpack command in the following format:

sbpack PROFILE_NAME APPID CONFIG_FILE

In the command above, PROFILE_NAMEΒ refers to the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file. TheΒ  APPIDΒ parameter specifies the identifier of the app on the Platform. Use the sameΒ  APPIDΒ you used in the initial conversion step. Finally, CONFIG_FILEΒ is theΒ sb_wdl_schema.yamlΒ in which you made app execution optimizations. The final command should be, for example:

sbpack sbpla-profile sevenbridges-division/wdl-project/test-app sb_wdl_schema.yaml

This pushes the modified app configuration to the Platform and creates a new revision (version) of the app. Once this is done, you are ready to run a task using the app.

Updating already converted and optimized apps

If you have already converted your app, made optimizations in the sb_wdl_schema.yaml file, and pushed the app to the Platform, all subsequent updates to the app's WDL code and the process of propagating the update to the Platform are quite straightforward. If the updates you made do not require changes to manually configured parameters in the sb_wdl_schema.yaml file (such as inputs, outputs, requirements, etc.), create a new code package by running a command in the following format:

sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --sb-schema SB_SCHEMA

This command is almost the same as the initial app conversion step, but differs in the additional --sb-schema argument. This argument allows you to provide and reuse an existing sb_wdl_schema.yaml configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on the Seven Bridges Platform. The command will generate a new code package based on your updated WDL code provided through --workflow-path and --entrypoint and the YAML or JSON configuration file provided through --sb-schema, and push the updated app to the Platform creating a new revision (version).

Copying WDL apps between projects on the Platform

When an app is on the Platform, you canΒ copyΒ it and use it on other Platform projects. To copy WDL apps between projects, use theΒ sbcopyΒ command that is a part of theΒ sbpackΒ utility:

sbcopy [--profile PROFILE] --appid APPID --projectid PROJECTID

The command takes the following arguments:

  • PROFILE: refers to the Seven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in theΒ Seven Bridges credentials file.
  • APPID: specifies the identifier of the app on the Platform.Β Takes the formΒ {user or division}/{project}/{app_id}. If you are usingΒ Enterprise, theΒ {user or division}Β part is name of your Division on the Platform; otherwise, specify your Platform username. TheΒ {project}Β part is the source project where the app is located andΒ {app_id}Β is the ID of the app you want to copy; for exampleΒ my-division/my-new-project/my-wdl-app.
  • PROJECTID: is the identifier of the destination project where the app will be copied.Β Takes the formΒ {user or division}/{project}. If you are usingΒ Enterprise, theΒ {user or division}Β part is name of your Division on the Platform; otherwise, specify your Platform username.
    The final command should be, for example:
sbcopy sbpla-profile sevenbridges-division/wdl-project/test-app other-division/my-wdl-project

πŸ“˜

Note that WDL app copies made through standard visual interface or API methods instead of using sbcopy will still point to the originally pushed code package and the original project where it is located. This might cause failures due to lack of permissions, if users who need to run the copied instances of the app aren't added to the project where the original code package is located. To avoid this, please use sbcopy to copy WDL apps between projects on the Platform, as described above.

sbpack_wdlΒ command reference

Here is a list describing all available arguments od theΒ sbpack_wdlΒ command that is used to convert and push WDL apps for execution on the Seven Bridges Platform.

ArgumentRequiredDescription
-h, --helpShows the list of all arguments and their corresponding explanations.
--profile PROFILESeven Bridges Platform profile containing the Platform API endpoint and authentication token, as set in the Seven Bridges credentials file. If you are using the default profile, this parameter can be omitted.
--appidΒ APPIDRequiredThe ID of theΒ WDL app once it is pushed to the Seven Bridges Platform. Takes the form {user or division}/{project}/{app_id}. If you are using Enterprise, theΒ {user or division}Β part is name of your Division on the Platform; otherwise, specify your Platform username. TheΒ {project}Β part is the project to which you want to push the app andΒ {app_id} is the ID you want to assign to the app, for exampleΒ my-division/my-new-project/my-wdl-app.
--workflow-path WORKFLOW_PATHRequiredPath to the main workflow directory (the local directory where the app's files are located).
--entrypoint ENTRYPOINTRequiredRelative path to the the file that contains the app'sΒ WDL code from the main workflow directory defined in --workflow-path.
--sb-package-id SB_PACKAGE_IDID of an already uploaded package. If you have already converted and pushed the app to the Platform, it has its own code package ID, as shown in theΒ code_package key in the sb_wdl_schema.yaml file. When the package ID is provided, the conversion script will skip the upload step and thus take less time to execute.
--sb-docΒ SB_DOCPath to the app description document written in Markdown. The document is meant to provide additional details about the app and will be shown when viewing app details on the Platform. If not provided, README.md will be used if available in the same directory where entrypoint file is located.
--womtool-input WOMTOOL_INPUTRequired ifΒ --womtool-path isn't usedPath to JSON inputs file containing your app's inputs schema generated using WOMtool.
--womtool-path WOMTOOL_PATHRequired ifΒ --womtool-input isn't usedPath to the WOMtoolΒ jarΒ file on your local machine. If you don't already have a JSON inputs file that you can provide via --womtool-input, this command will use WOMtool to generate the file on the fly and provide it toΒ sbpack_wdl automatically.
--sb-schema SB_SCHEMAPath to an existing sb_wdl_schema file in JSON or YAML format. This allows you to use an existing configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on the Seven Bridges Platform.
--dump-sb-appDumps the converted app to a local file without pushing it to the Platform. Using this option will enable you to convert the app and generate theΒ sb_wdl_schema.yaml Β file to make configuration optimizations prior to pushing the app the Platform.
--no-packageReferences a git repository containing the app's code instead of pushing the app's code to the Platform.Β Git repository address is specified as the git_pull key in the sb_wdl_schema.yamlΒ file, instead ofΒ code_package. For example:Β git_pull: https://git.domain.com/repository. The value for theΒ git_pull key is the URL you would normally use to clone the repository to your local environment.
--jsonCreates the sb_wdl_schema file in JSON format instead of the default YAML.

Important notes for executing WDL apps on the Seven Bridges Platform

  • Workflows are executed in Local mode. Make sure your workflow can run in Local mode before porting it to the Seven Bridges Platform.
  • Use of Docker is required. See how to create and upload a Docker image containing your app and make sure to edit the WDL code to use the newly created image. If the Docker image is not specified for a process, a default alpine image will be used.
  • Execution is done in a dedicated working directory and all the WDL work is done in the wdl_workdirΒ directory inside the working directory. Avoid using and relying on hard-coded paths in workflows.

Differences between running CWL and WDL apps on the Platform

Executions on the Platform normally result in separate jobs (steps) being created for each tool in the workflow. When a WDL pipeline is executed on the Platform, a single executionΒ job will be created, regardless of the number of tools within the pipeline. This "one app, one job" approach results in the following differences compared to CWL app executions:

  • Memoization is not available for WDL apps. As memoization relies on using previous job outputs to skip identical jobs in new executions, it is not useful in situations when there is only one job in an app execution.
  • Handling ofΒ spot instanceΒ interruptionΒ can't be used withΒ WDL apps. If a spot instance gets terminated when a WDL app is running, the app would have to be rerun on an on-demand instance from the beginning, which would result in increased costs instead of savings. Therefore, it is recommended to disable spot instances for WDL app executions.
  • Task stats and logsΒ are organized differently. As task statistics and logs are usually shown and organized per job, WDL apps will have cumulative stats for the single job in the execution, while logs will be saved in a single folder.