Config files and automation settings

The Seven Bridges Platform automation example from the previous section had several hard-coded values (literals) in it, for example the ID of the public reference project or the ID of the automated app. This is not recommended, because the meaning of literals is not always intuitively clear and, if repeated many times, literals create error-prone redundancy in the automation code. In addition, if literals need to change, the automation source code itself needs to be changed.

It is considered best practice to put literals as settings into automation config files. This way, literals are assigned an expressive name and managed at a central place. In addition, should this need arise, every automation setting inside a config file can be overridden at runtime when starting the automation, without changing the automation code. In the automation code, automation settings can be conveniently accessed from within steps using the predefined self.config_ member variable.

Below is how a config file (in YAML format) for the automation example of the previous section can look like.

public_project_id: admin/sbg-public-data
fastqc: admin/sbg-public-data/fastqc-0-11-4

The name of the config file is not important, but it needs to be either in JSON or YAML format and must be located inside subdirectory configs that is located in the automation root directory.

The configs directory can contain more than one configuration file, in which case all of them are read and used by the automation.

The revised automation code example from the previous section that now utilizes these new settings looks like this:

from freyja import Step, Automation, Input
from hephaestus import SBApi
from hephaestus.steps import (
    FindOrCreateProject,
    FindOrCopyFilesByName,
    FindOrCopyApp,
    FindOrCreateAndRunTask
)
 
 
class Main(Step):
    project_name = Input(str)
 
    def execute(self):
 
        public_data_project = SBApi().projects.get(self.config_.public_project_id)
 
        my_project = FindOrCreateProject(name=self.project_name).project
 
        fastq_file = FindOrCopyFilesByName(
            names=["example_human_Illumina.pe_1.fastq"],
            from_project=public_data_project,
            to_project=my_project,
        ).copied_files[0]
 
        fastqc_app = FindOrCopyApp(
            app_id=self.config_.fastqc,
            to_project=my_project
        ).app
 
        task = FindOrCreateAndRunTask(
            inputs={"input_fastq": [fastq_file]},
            app=fastqc_app,
            in_project=my_project,
        ).finished_task
 
        print(task.outputs)
 
 
if __name__ == "__main__":
    Automation(Main).run()

Implicit settings when running on the Seven Bridges Platform

When submitting automation runs on the Seven Bridges Platform, the automation run information that can sometimes be useful is provided through implicit automation settings:

• The current automation run ID is available with self.config_.sb_automation_run_id .

Secret settings

Some automation settings stored in config files should never be exposed to anyone but yourself. Examples for such secret settings include your personal Seven Bridges authentication token or a user password that authenticates you against an external service, for example an email gateway.

Starting with version 0.16.2, Freyja recognizes a special config file named 'secrets.yaml' in which you should put all your secret settings, if you have any. Settings in this file are used when you run the automation locally, but do not become part of the input-output schema that is stored alongside your code package file on the Seven Bridges Platform. The location of this special config file is the same as for all your other config files inside the 'configs' subdirectory. Make sure that when you have a secret setting that you put it inside 'secrets.yaml'.

🚧

Besides putting all your secret settings inside secrets.yaml, you also need to make sure that this special config file does not become part of the automation code package file that you later create and submit to the Seven Bridges Platform.

The best way to ensure this is to include configs/secrets.yaml inside .packignore. See Deploy and run automations on the Seven Bridges Platform for further details on how code package files are created and uploaded.

🚧

If you use a shared source code management systems, also make sure to exclude this file from submission into this repository. If you use Git, then you should list configs/secrets.yaml inside .gitignore as well.

Overriding automation settings at runtime

Every automation setting inside a config file can be changed at runtime without modifying the config file or the automation code. You can do this both when running the automation locally and when running on the Seven Bridges Platform.

For example, if you wanted to run revision 18 instead of the latest revision of FastQC, this is how you could do it from your local command line without changing the automation code itself:

FREYJA_CONFIG_fastqc=admin/sbg-public-data/fastqc-0-11-4/18 python fastqc_with_config.py run --project_name my_project

As an alternative, you can also provide one or more --config arguments followed by a config filename to pass runtime settings to your automation.

Automation settings can also be overridden when starting automations on the Platform. To do this, use the --settings argument of the 'sb automations start' command of the SB CLI.

Next: App-executing steps