The Seven Bridges Knowledge Center

The Seven Bridges Platform is a simple solution for doing bioinformatics at industrial scale. But sometimes, everyone needs a little help.

Get Started

Deploy and run automations on the Seven Bridges Platform

Deploying and running automations on the Seven Bridges Platform has many advantages, including:

  • ability for automation users to run an automation without having to install it first on their own computer
  • instant access to all versions of the automation, including the most recent and all previous ones
  • access to automation run history with the ability to quickly re-run a previously run automation
  • quick access to all automation inputs, settings, code, and logs for developers to debug a failed automation run

In order to run an automation on the Seven Bridges Platform, the automation source code needs to be first compressed into a code package file (.zip format). This code package file must then be uploaded to the Seven Bridges Platform.

What follows is a step-by-step guide how to do this. It requires Freyja version 0.16.3 or higher and SB CLI version 0.12.3 or higher.

Project structure

Before creating a code package file for execution on the Seven Bridges Platform, we want to make sure that the Python project containing the automation source code has the correct structure. This is important, because the Seven Bridges execution service relies on the existence of specific directories and files.

Here is an example structure of a project named 'project_1' that is compatible with execution on the Seven Bridges Platform:

project_1
|-- config
|   `-- secrets.yaml      # configuration file(s) (optional)
... ...                   #     more config files (optional)
|-- project1              # automation main Python package (required)
|   `-- __main__.py       # module with automation main step (required)
... ...                   # other code for your automation (optional)
... ...                   #     as files or packages
|-- .entrypoint           # contains name of main Python package (required)
|-- .packignore           # files and directories to exclude from packaging (optional)
|-- requirements.txt      # Python dependencies of this automation (required for platform execution)
|-- README.md             # readme (optional)
`-- setup.py              # setup.py (optional)

You can use the command 'freyja init --app project1' inside a directory named 'project_1' to create a project with the structure shown above. Using the 'init' command makes sure that the project structure is compatible with later execution on the Seven Bridges Platform. Don't forget to upgrade to the latest version of Freyja if you have an older version installed.

To understand how to best set up a repository that contains multiple automations, please refer to the section Repository structure.

.entrypoint file

The file .entrypoint contains the name of the Python package that holds the automation main module with the automation main step.

For the above example, the entrypoint file looks as follows:

[email protected]:~/project_1$ cat .entrypoint
entrypoint: project1

.packignore file

The file .packignore lists all files and directories that should not become part of the code package file that is created later for platform submission (see below).

At a minimum, this file should mention your secret settings file (config/secrets.yaml) if you have one, but typically also lists other locally generated files not required for Platform execution.

[email protected]:~/project_1$ cat .packignore
/configs/secrets.yaml
__pycache__/
automation.log
state.json

Requirements file

The file requirements.txt lists all Python libraries that need to be installed in order for the automation code to execute successfully. At the very least, it needs to declare the dependency of Hephaestus library, including version information. This ensures that the right version of library is used for running the automation, even after a new version of Hephaestus has been released.

[email protected]:~/project_1$ cat requirements.txt
freyja==0.16.2
hephaestus==0.14.3

Please make sure that Hephaestus and Freyja versions in your requirements.txt file match those that you have installed and tested with on your local system. This prevents platform execution failures due to library version differences.

Further note that if Freyja is not listed as requirement, Hephaestus automatically pulls the newest version of Freyja as dependency. Because until Freyja version 1.0 backward compatibility is not guaranteed, the safer option is to explicitly require the installation of a specific version of Freyja that you know is compatible with your automation and list it before Hephaestus within requirements.txt.

Advance access

The ADK depends on some advance access (i.e. early access) features of the Seven Bridges API, for example to import/export files from/to volumes.

To make sure your automation script executes successfully when run on the Seven Bridges Platform even in presence of advance access API calls, include the following setting in one of your config files:

[email protected]:~/project_1$ cat configs/project1.yaml
sb_api_advance_access: True

Creating code package file

The code package file is a .zip file that contains all source code required to run the automation. To create this file, you can run the freyja build command from inside your project root directory. The earlier freyja pack command is now deprecated.

[email protected]:~/project_1$ freyja build
Code package file created successfully at /home/rosalind/project_1/project_1.zip

To check the content of this zip:

[email protected]:~/project_1$ unzip -l project_1.zip
Archive:  project_1.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      290  2020-04-24 12:15   README.md
      246  2020-04-24 13:44   .schema
       61  2020-04-24 12:15   .packignore
       15  2020-04-24 12:15   requirements.txt
       21  2020-04-24 12:15   .entrypoint
        0  2020-04-24 13:44   project1/
      590  2020-04-24 13:44   project1/__main__.py
        0  2020-04-24 12:15   configs/
---------                     -------
     1223                     8 files

Note that the zip file and your project now contains an additional file .schema. This file contains a formal description of all automation inputs, outputs, and settings and is automatically inferred from your automation's main step and config files.

The Seven Bridges Platform uses this schema information to properly display automation inputs and outputs on the visual interface.

[email protected]:~/project_1$ cat .schema
{
  "inputs": {
    "file_name": {
      "type": "String",
      "meta": {
        "ui_type": "text"
      },
      "required": true,
      "default": null,
      "description": null
    }
  },
  "outputs": {},
  "settings": {},
  "secrets": {}
}

🚧

Important note

Before you submit any code package to the Seven Bridges Platform, make sure neither files inside your code package nor the schema contain confidential information, for example user tokens or private passwords.

You can use secret settings and .packignore (see above) to prevent confidential information from leaking into the generated code package.

Using the command freyja schema you can dump the current input-output schema for your automation on your console without actually creating the code package file. This is useful for debugging in case schema generation fails with some error.

Creating automation entity

Code packages are organized under automation parent entities. Each automation parent entity can have one or more code packages. Think of code packages as different versions of an automation.

📘

If your goal is to deploy another code package for an already existing automation entity, you can skip this section and proceed below with creating a new code package entity.

A new automation entity can be created with the sb automations create command. If you are not operating inside an Enterprise Division that already has a billing group set up, this command allows you to specify of a billing group. if omitted, your default billing group is used.

📘

Compute and storage cost of automations

The billing group of your automation entity is charged for storage cost incurred by code package files stored inside that automation (see below).

Seven Bridges does not charge for compute cost incurred by running the automation Python code itself. However, if your automation executes tasks inside a Seven Bridges project, then the billing group of that project is charged for the compute cost of these tasks.

To find billing group UUID, use the following command (doesn't apply to Enterprise users):

[email protected]:~$ sb billing list
'1b1b275a-XXXX-XXXX-XXXX-XXXX92c1ad4c'  'My billing group'

To create new automation entity named project_1, the following command is used (--billing-group is not required for Enterprise users). Check command synopsis for additional options (sb automations create --help).

[email protected]:~$ sb automations create --name project_1 --billing-group 1b1b275a-XXXX-XXXX-XXXX-XXXX92c1ad4c
d2d22a13-2923-XXXX-XXXX-6a0ca56cf057    project_1       rosalind_franklin   rosalind_franklin   2019-03-15T15:30:49Z    rosalind_franklin   2019-03-15T15:30:49Z

The first column is the UUID of the newly created automation entity. This UUID is required in the next step when we create the code package entity under this automation entity.

To get a list of all automation entities currently available for your user, you can use the sb automations list command:

[email protected]:~$ sb automations list
d2d22a13-2923-XXXX-XXXX-6a0ca56cf057 project_1 rosalind_franklin 2019-03-15T15:30:49Z 2019-03-15T15:30:49Z rosalind_franklin

To see all details for the newly created automation entity, use the sb automations get command. To make the output more readable, we can requested JSON output format and pass it through json.tool to get JSON in a pretty format.

[email protected]:~$ sb --output json automations get d2d22a13-2923-XXXX-XXXX-6a0ca56cf057 | python -m json.tool
{
"id": "d2d22a13-2923-XXXX-XXXX-6a0ca56cf057",
"name": "project_1",
"owner": "rosalind_franklin",
"created_by": "rosalind_franklin",
"created_on": "2019-03-15T15:30:49Z",
"modified_by": "rosalind_franklin",
"modified_on": "2019-03-15T15:30:49.205106Z"
}

Adding automation members

If you want other users to run this automation they must be added as members to the automation entity.

[email protected]:~$ sb automations members create --automation-id d2d22a13-2923-XXXX-XXXX-6a0ca56cf057 --user "francis_crick" --read --copy --execute

Minimum permissions to successfully run an automation are READ, COPY, and EXECUTE. Only ADMIN members of an automation can add or remove other members to or from an automation.

If you operate inside a Division, don't forget to prefix the username with the division name, e.g. my_division/francis_crick.

To see all current members of your automation, type:

$ sb automations members list --automation-id d2d22a13-2923-XXXX-XXXX-6a0ca56cf057
rosalind_franklin   copy,execute,admin,write,read
francis_crick   execute,read,copy

Creating code package entity

Each automation parent entity can hold one ore more automation code package entities. Think of each code package entity as a distinct version of an automation. Each code package entity is associated with exactly one code package file.

Use the following command to create a new code package entity with version named '0.0.1'. This command will automatically upload the code package file project_1.zip created above.

Check command synopsis for additional options (sb automations packages create --help). You will need at least WRITE permission on the automation parent entity for this command to be successful.

[email protected]:~$ sb automations packages create --automation-id d2d22a13-2923-XXXX-XXXX-6a0ca56cf057 --version 0.0.1 --file 5c8bc26ce4b0acf7a81346a6
7a85b850-c93e-XXXX-XXXX-1a6b31094691    d2d22a13-2923-XXXX-XXXX-6a0ca56cf057    0.0.1   5c8bc26ce4b0acf7a81346a6    rosalind_franklin   2019-03-15T15:40:05Z

Hint: You can use the file ID of your uploaded code package (in this example 5c8bc26ce4b0acf7a81346a6) to later retrieve the code package file back from the Platform. To do this, you can use the following command:

[email protected]:~$ sb download --file 5c8bc26ce4b0acf7a81346a6
Downloading file `project_1.zip` with ID `5c8bc26ce4b0acf7a81346a6`.
1.99 KiB / 1.99 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 10.74 KiB/s 0s
'success' '5c8bc26ce4b0acf7a81346a6' 'project_1.zip` '2035'

Start new automation run

The automation is now ready to be executed on the Seven Bridges Platform. We use the sb automations start command to create new automation runs.

The following command starts the most recently created code package for the automation named 'project_1'. If you want to execute a specific code package, one possibility is to use argument --package-id instead of --automation-name. Check command synopsis for additional options (sb automations start --help).

[email protected]:~$ sb automations start --automation-name project_1 --inputs '{"project_name":"my first automation"}'
d708b346-90af-4923-85d2-32ef61011a53    project_1 - 0.0.1 - 03-15-19 15:52:43   project_1   0.0.1   project_name=my first automation        2019-03-15T15:52:43Z                rosalind_franklin   QUEUED_FOR_EXECUTION

The first column of the command output is the UUID of the newly created automation run. In the last column of the command output we can see that the run is currently in status 'QUEUED_FOR_EXECUTION' and it should start running momentarily.

In this example, the automation had only a single required input (project_name). If your automation has additional inputs, they can be added to the input string in JSON notation.

We can use sb automations runs list to get a list of all automation runs with their status, or use the run UUID to obtain all details about a specific automation run.

[email protected]:~$ sb --output json automations runs get d708b346-90af-4923-85d2-32ef61011a53 | python -m json.tool
{
    "id": "d708b346-90af-4923-85d2-32ef61011a53",
    "name": "project_1 - 0.0.1 - 03-15-19 15:52:43",
    "automation": {
        "href": "https://api.sbgenomics.com/v2/automation/automations/d2d22a13-2923-44b7-8908-6a0ca56cf057",
        "id": "d2d22a13-2923-XXXX-XXXX-6a0ca56cf057",
        "name": "project_1",
        "owner": "rosalind_franklin",
        "created_by": "rosalind_franklin",
        "created_on": "2019-03-15T15:30:49.000Z",
        "modified_by": "rosalind_franklin",
        "modified_on": "2019-03-15T15:30:49.205106"
    },
    "package": {
        "id": "7a85b850-c93e-XXXX-XXXX-1a6b31094691",
        "automation": "d2d22a13-2923-XXXX-XXXX-6a0ca56cf057",
        "version": "0.0.1",
        "location": "5c8bc26ce4b0acf7a81346a6",
        "created_by": "rosalind_franklin",
        "created_on": "2019-03-15T15:40:05.710Z"
    },
    "inputs": {
        "project_name": "my first automation"
    },
    "settings": null,
    "created_on": "2019-03-15T15:52:43.935Z",
    "start_time": "2019-03-15T15:52:48.164Z",
    "end_time": "2019-03-15T16:01:44.046Z",
    "created_by": "rosalind_franklin",
    "status": "FINISHED",
    "execution_details": {
        "log_file": {
            "href": "https://api.sbgenomics.com/v2/files/5c8bcc64e4b09d72f28d0147",
            "id": "5c8bcc64e4b09d72f28d0147",
            "name": "automation.log",
            "project": "rosalind_franklin/automations",
            "created_on": "2019-03-15T16:01:40Z",
            "modified_on": "2019-03-15T16:01:40Z",
            "type": "file",
            "parent": "5c8bcc64e4b09d72f28d0144",
            "size": 26024,
            "origin": {},
            "storage": {
                "type": "PLATFORM"
            }
        },
        "state_file": {
            "href": "https://api.sbgenomics.com/v2/files/5c8bcc66e4b09d72f28d014c",
            "id": "5c8bcc66e4b09d72f28d014c",
            "name": "state.json",
            "project": "rosalind_franklin/automations",
            "created_on": "2019-03-15T16:01:42Z",
            "modified_on": "2019-03-15T16:01:42Z",
            "type": "file",
            "parent": "5c8bcc64e4b09d72f28d0144",
            "size": 28771,
            "origin": {},
            "storage": {
                "type": "PLATFORM"
            }
        }
    }
}

This outputs a lot of information on the automation run, including updated automation status (in this case the automation run has already FINISHED successfully).

Please refer to the Seven Bridges Automation CLI for further information on available automation commands, including commands for automation membership management and how to check the automation execution log in case of errors.

Updated 24 days ago

Deploy and run automations on the Seven Bridges Platform


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.