The Seven Bridges Knowledge Center

The Seven Bridges Platform is a simple solution for doing bioinformatics at industrial scale. But sometimes, everyone needs a little help.

Get Started

Metadata on the Seven Bridges Platform

Metadata associated with files makes them searchable, keeping your file collection manageable as it grows. It also enables them to be properly grouped for analyses.

Overview

One of the most common reasons for failed tasks is missing or improper metadata. Lack of proper metadata increases the chances that tools and workflows will fail to run correctly. We recommend that you set the metadata fields for your input files before executing analyses on the Platform.

There are three ways to enter or change file metadata:

We have 14 metadata fields associated with each file on the Platform. These are subdivided into three categories (File, Sample, and General). The recommended practice is to enter as much metadata as possible when you first upload files to the Platform. For instance, for raw sequencing files, you should enter Platform (sequencing platform) and Sample ID. Of these fields, there are seven metadata fields that we highly suggest you set for your data. While your tasks may run correctly without them, these metadata fields will help optimize your analyses. These fields are labeled in the table below with a red suggested tag in the Name column.

See the tables below for more details about the metadata fields.

Metadata categories

File

In the following table, you will find the name, description, and values of metadata fields for File. The second column, API key, allows you to access the specified metadata field through the API. Learn more about accessing metadata via the API.

There are six metadata fields that we highly suggest you set for your data. While your tasks may run correctly without them, these metadata fields will help optimize your analyses. These fields are labeled in the table below with a red suggested tag in the Name column.

Name
API key
Description
Values

Library ID
suggested

library_id

This is an identifier for the sequencing library preparation.

The value set in this field does not affect whether or not the workflow runs successfully. However, all files that come from the same sequencing library must have the same value.

The Library ID will be written to the read group tag (@RG:LB) in SAM or BAM files. All aligner apps are programmed to add RG fields to the aligned BAM according to the Library ID.

This takes a string.

Platform
suggested

platform

This is the version (manufacturer, model, etc.) of the technology that was used sequencing or assaying.

Only some tools and workflows may require a value for the Platform field. However, it is recommended that you set it whenever possible, unless you are certain that your workflow will work without it.

This takes a string.
Suggested values:

  • Illumina HiSeq
  • Illumina GA
  • ABI capillary sequencer
  • Illumina MiSeq
  • ABI SOLiD
  • Ion Torrent PGM
  • LS 454
  • Illumina HiSeq X Ten
  • Illumina
  • Helicos
  • PacBio
  • Not available

Platform unit ID
suggested

platform_unit_id

This is an identifier for lanes (Illumina), or for slides (SOLiD) in the case that a library was split and ran over multiple lanes on the flow cell or slides. The Platform unit ID refers to the lane ID or the slide ID.

The value supplied in the Platform unit ID field will be written to the read group tag (@RG:PU) in SAM or BAM files. All aligner apps add read group fields to the aligned BAM file on the basis of Platform unit ID metadata.

This takes a string.

Paired end
suggested

paired_end

For paired-end sequencing, this value determines the end of the fragment sequenced.

For paired-end read files, this field indicates whether the read file is left end or right end. Set ‘1’ for left end and ‘2’ for right end reads. This is used to group pairs. If the FASTQ file is a single-end read this field should be left as ‘-’.

Note: It is important for two members of paired-end reads to have identical Sample ID, Library ID, Platform unit ID, and File segment number.

This takes a value of 1 or 2.
Note: For single-end sequencing no value is needed

File segment number
suggested

file_segment_number

If the sequencing reads for a single library, sample and lane are divided into multiple (smaller) files, the File segment number is used to enumerate these. Otherwise, this field can be left blank.

This information can be used for batching when processing files with a workflow.

This takes an integer.

Quality scale
suggested

quality_scale

For raw reads, this value denotes the sequencing technology and quality format. For BAM and SAM files, this value should always be ‘Sanger’.

Enter this value for all FASTQ files, unless they are used in a workflow with a FASTQ quality scale detector wrapper.

Choose from one of the following options:

  • sanger
  • llumina13
  • illumina15
  • illumina18
  • solexa
    Or, enter no value.

Experimental strategy

experimental_strategy

This is the method or protocol used to perform the laboratory analysis.

This takes a string.
Suggested values:

  • DNA-Seq
  • WXS
  • WGS
  • Amplicon
  • Bisulfite-Seq
  • RNA-Seq
  • miRNA-Seq
  • Total RNA-Seq
  • Not available

Reference genome

reference_genome

The reference assembly (such as HG19 or GRCh37) to which the nucleotide sequence of a case can be aligned.

This takes a string.
Suggested values:

  • human_g1k_v37
  • human_g1k_v37_decoy
  • ucsc.hg19
  • Homo_sapiens.Ensembl.GRCh37
  • Homo_sapiens.GRCh38.dna.primary_assembly
  • ion_torrent.hg19
  • mouse_mm9_ucsc
  • ens_mouse_mm9_genome
  • mouse_mm10_ucsc

Sample

In the following table, you will find the name, description, and values of metadata fields for Sample. The second column, API key, allows you to access the specified metadata field through the API. Learn more about accessing metadata via the API.

Name
API key
Description
Value

Sample ID
suggested

sample_id

A human readable identifier for a sample or specimen, which could contain some metadata information. A sample or specimen is material taken from a biological entity for testing, diagnosis, propagation, treatment, or research purposes, including but not limited to tissues, body fluids, cells, organs, embryos, body excretory products, etc.

Tools use Sample ID to separate files that come from different samples.
For SAM and BAM files, the value supplied in the Sample ID field is written to the read group tag (@RG:SM). All aligners add read group fields to the aligned BAM file using the file’s Sample ID metadata.

This takes a string.

Sample type

sample_type

The type of material taken from a biological entity for testing, diagnosis, propagation, treatment, or research purposes. This includes tissues, body fluids, cells, organs, embryos, body excretory products, etc.

This takes a string.
Suggested values:

  • Blood normal
  • Tumor tissue
  • Normal tissue
  • Primary cells
  • Stem cells
  • Embryo
  • Cell Line
  • Saliva
  • Control
  • Not available

General

In the following table, you will find the name, description, and values of metadata fields for General. The second column, API key, allows you to access the specified metadata field through the API. Learn more about accessing metadata via the API.

Name
API key
Description
Value

Investigation

investigation

A value denoting the project or study that generated the data.

This takes a string.

Species

species

A group of organisms having some common characteristic or qualities, that differ from all other groups of organisms and that are capable of breeding and producing fertile offspring.

This takes a string.
Suggested values:

  • Homo sapiens
  • Mus musculus

Batch number

batch_number

This is an assigned distinctive alpha-numeric identification code that signifies grouping.

This takes an integer.

Case ID

case_id

This is a human-readable identifier, such as a number or a string for a subject who has taken part in the investigation or study.

This takes a string.

Apart from the standard set of metadata fields that can be seen through the visual interface, custom metadata fields can be added via the command line uploader or via the API.

Grouping and distinguishing files by metadata

Some apps group and process files based on their metadata. In particular, apps that can process multiple files in parallel will do so in groups of files that have the same metadata value(s) for certain metadata fields. These fields are: Sample ID, Library ID, Platform unit ID, and File segment number.

There is a prioritization rule governing how files can be grouped on the basis of metadata: files are grouped by their value for a metadata field only if all values hierarchically above that given field match. Files are grouped following the hierarchy listed below:

  1. Sample ID
  2. Library ID
  3. Platform unit ID
  4. File segment number

For example, files batched by Library will first be sorted by Sample ID with a secondary sort by Library ID. Consider the files described in the table below:

File name
Sample ID
Library ID

File 1

A

1

File 2

A

1

File 3

B

2

File 4

B

3

File 5

B

2

File 6

B

3

These six files will be sorted into 3 groups:

  • Sample ID "A" with Library ID "1" (File 1 and File 2)
  • Sample ID "B" with Library ID "2" (File 3 and File 5)
  • Sample ID "B" with Library ID "3" (File 4 and 6)

Learn more about grouping files in batch analysis.

Metadata is also used to distinguish files. The rule governing this is that no two files can have the same metadata values for all fields. To use a common example, paired end read FASTQ files will have the the same metadata values for the fields Sample ID through to File segment number, in the list above. However, they must take different values for the Paired end field.

Remember, it is important for two members of paired-end reads to have identical Sample ID, Library ID, Platform unit ID, and File segment number.

Metadata can often be built into the filename. For instance, files produced by Illumina sequencers follow the naming schema: <sample name>_<barcode sequence>_L<lane>R<read number><set number>.fastq.gz

Consider the following two paired end files generated by an Illumina sequencer:

  • NA10831_ATCACG_L002_R1_001.fastq.gz
  • NA10831_ATCACG_L002_R2_001.fastq.gz

In this example, we can read off some metadata values from the filename: the Sample ID of both files is 'NA10831', their Platform unit ID value is 002, and their Paired end values are 1 and 2 respectively. These values can then be inputted into the Platform.