The Seven Bridges Knowledge Center

The Seven Bridges Platform is a simple solution for doing bioinformatics at industrial scale. But sometimes, everyone needs a little help.

Get Started    What's new?

Upload via the command line

Large datasets hosted on corporate or academic clusters and workstations can be uploaded to the Seven Bridges Platform using the Seven Bridges Command Line Uploader. This is a fast and secure upload client that has been optimized to efficiently upload files to the Seven Bridges Platform, taking advantage of parallelization where possible.

The Seven Bridges Platform command line uploader requires Java 1.7 or newer. Issue the following command in a terminal window.

$ java -version

and look for the version number in the first line of the output. It should look something like this:

java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)

Install the command line uploader

To install the Seven Bridges Platform Command Line Uploader:

This feature depends on your cloud infrastructure provider

To use this feature, you should know the cloud infrastructure provider on which you run the Seven Bridges Platform: Amazon Web Services in the US (AWS US East); Amazon Web Services in Frankfurt, Germany (AWS EU); or Google Cloud Platform.

If you didn't chose a cloud provider when you signed up for the Platform, you are using AWS US East. If you signed up from early 2016, you had the option to select your cloud provider from AWS and Google Cloud Platform.

If you signed up from early 2017, you had the additional option to select AWS EU as your cloud provider.

  1. Download the Command Line Uploader:

a. Seven Bridges users on Amazon Web Services US (AWS US East)
b. Seven Bridges users on Amazon Web Services EU (AWS EU)
c. Seven Bridges users on Google Cloud Platform (GCP)

  1. Unpack the uploader to a directory of your choice. Your home directory is a good default location. To do this, enter:
$ tar zxvf ~/Downloads/sbg-uploader.tgz -C ~
  1. Run the uploader with the -h switch to list all the available command line options (listed below) and their usage:
$ ~/sbg-uploader/bin/sbg-uploader.sh -h

Command line options

sbg-uploader.sh [-h] [-l] [-p id] [-t token] [-u username] [-x url] file [--tag "enter tag here"]
Option
Description

-h
--help                                   

This option prints a short usage summary and will cause the uploader to ignore any other options. It will exit with a status of 0 (see Exit Statuses below).

-l
--list-projects

This option gives you a list of projects available as upload targets with the following two columns:

  • The project identifier which is used for specifying the target project (e.g. rfranklin/samtools where "rfranklin" is the project owner and "samtools" is the name of the project).
  • The project name (e.g. “my new project”).

-p <id>
--project <id>

This option is used for choosing the target project by specifying its unique identifier, e.g. username/projectname.

Use the --list-projects option to find the project identifier for your project.

To upload files to a project, you must be a member of that project and must have the write permission granted by the project administrator.

This option is mandatory.

-t <token>
--token <token>

This option is used to specify an authorization token. This option overrides the credentials read from the configuration file (see below).

-u <username>
--user <username>

This option is used to specify a username for the Seven Bridges Platform. If omitted and not using the --token option, you will be prompted for a username.
This option can be used only by users who haven't connected their Seven Bridges Platform account with an eRA account.

-x <url>
--proxy <url>

This option specifies a proxy server through which the uploader should connect. The proxy parameter should be of the form proto://[username:password@]host[:port]. proto can be 'http’ or 'socks’. HTTP proxies must allow the CONNECT command to port 443. SOCKS proxies can be both SOCKS4 and SOCKS5.

  • username and password are optional and will be used if the proxy requires authentication.
  • host is required.
  • port is optional. If omitted, the uploader will use 8080 for HTTP and 1080 for SOCKS proxies.

--tag

Use this option to enter tags composed of strings for your files. Format your tags as such: --tag "first tag here" --tag "second tag here" --tag "third tag here"

--list-tags

Use this option to list all the tags in your destination project.

-mm
--manifest-metadata

Use this option to upload multiple files and set their metadata using the manifest file.

To only apply individual metadata fields from the manifest, list them after the --manifest-metadata option, e.g. --manifest-metadata sample paired_end.

This option should be used in combination with the --manifest-file option.

-mf
--manifest-file

Use this option to specify the name of the manifest file.

This option should be used in combination with --manifest-metadata , e.g. --manifest-metadata --manifest-file filename.csv.

To only upload files while omitting the metadata use --manifest-file filename.csv.

--dry-run

Use this option to only output data in the terminal and check the settings without uploading anything.

To output information about specific metadata fields, list them after the --dry-run option, e.g. --dry-run sample library.

Tagging files

You can also tag files as you upload them via the Seven Bridges Uploader or from an FTP or HTTP(S) server.

Learn more about tagging your files in general on the Platform and how tags are beneficial to organizing your data.

Only the following characters are allowed in file names:

  • alphanumeric characters (lowercase and uppercase letters of the English alphabet and numbers 0 - 9),
  • underscore (_),
  • dash (-),
  • dot (.).

Upload multiple files

To upload larger volumes of data, use the command line uploader with a manifest file.

sbg-uploader.sh -u username --manifest-file filename.csv

Authentication

Getting the authentication token

You can obtain an authentication token for your Seven Bridges Platform account from the Developer Dashboard.

The Seven Bridges Platform Command Line Uploader looks for credentials in the following locations in order:
If the -u username option is given, the uploader prompts for and reads the password from standard input.

  1. If the -t token option is given, the uploader uses your authentication token.
  2. The uploader looks for an authentication token in the configuration file (see below).
  3. The uploader looks for the username and password in the configuration file (see below).
  4. The uploader prompts for and reads the username and password from standard input.

Configuration files

To avoid providing your credentials each time you use the uploader, you can store them in a configuration file. This file is called .sbgrc and resides in your home directory. This location varies across operating systems, but would typically be:

/home/$USER/.sbgrc on UNIX;
/Users/$USER/.sbgrc on OS X;
C:\Documents and Settings\%USERNAME% on Windows XP, 2000 and 2003; and
C:\Users\%USERNAME% on Windows Vista, 7, 8 and 10.

The .sbgrc configuration file should contain key-value pairs of the following form:
username = johndoe
password = supersecret123
auth-token = ec43d6dce3c54193ac18e3855f734ccf

You can specify the username and password, or the authentication token, or both. If both are given, then the authentication token will take precedence. The uploader will use these values only if no other authentication options are provided on the command line.

Please keep in mind that .sbgrc configuration file may only contain a single set of credentials. If multiple "username", "password" or "token" lines are encountered, the uploader will disregard all values but the last.

Metadata

Files on the Seven Bridges Platform are accompanied by metadata describing, amongst other things, their file type, origin, sample ID, and information about the sequencing technology used to create it. This metadata is often required by tools and workflows, and must be set before a file becomes fully usable. Click here to learn how to set metadata using the command line uploader. You can also manually set metadata later.

Exit statuses

Code
Status

0

Normal termination. The upload has either finished successfully, or usage information was written to standard output.

100

The upload has failed in the pre-processing phase or the uploader was unable to initialize it properly.

101

Input arguments were not properly set.

102

Mandatory options were not set.

103

Authentication error; invalid user credentials were used.

104

Bad metadata file.

200

Abnormal termination; an unknown error caused the upload to fail.

For example, if you want to use the Command Line Uploader to upload the FASTQ file sample1.fastq, with the associated metadata file, sample1.fastq.meta, to a project whose ID is 1234. Then, you should enter

sbg-uploader$ bin/sbg-uploader.sh -t $AUTH_TOKEN -p 1234 sample1.fastq --tag "fastq" --tag "sample 1"

and replace $AUTH_TOKEN with your own authentication token.

As shown in the example above, don't forget to change directory to the one containing the sbg-uploader, and to prefix the executable name with bin/.

Upload via the command line