The Seven Bridges Knowledge Center

The Seven Bridges Platform is a simple solution for doing bioinformatics at industrial scale. But sometimes, everyone needs a little help.

Get Started

Exporting files to a volume

📘

When exporting a file from the Platform to an attached volume, export is possible only to a volume that is in the same location (cloud provider and region) as the project from which the file is being exported.

The objective here is to export files from a Seven Bridges project to a volume (cloud bucket). Please note that export to a volume is available only via the API, including API client libraries, and through the Seven Bridges CLI.

Again, Seven Bridges bulk API calls should be used to reduce the overall number of API calls. Note that below examples make use of the copy_only export feature, which requires advance_access to be activated when initializing the API.

Not optimized for rate limit

In this example, files are fetched and exported in a loop, one by one.

for name in source_file_names:
     
    f = api.files.query(project=src_project, names=[name])[0]
     
    export = api.exports.submit_export(
        file=f,
        volume=volume,
        location="christian_demo_files/" + f.name,
        overwrite=True,
        copy_only=False
    )

Optimized for rate limit

Fetch and export files in bulk.

import time

def bulk_export_files(files, volume, location, overwrite=True, copy_only=False, chunk_size=100):
    "Exports list of files to volume in bulk"
 
    def is_running(response):
        if not response.valid:
            raise Exception(
                '\n'.join([
                    str(response.error),
                    response.error.message,
                    response.error.more_info
                ]))
        return response.resource.state not in ["COMPLETED", "FAILED", "ABORTED"]
     
    final_responses = []
 
    # export files in batches of 100 files each
    for i in range(0, len(files), chunk_size):
         
        # setup list of dictionary with export requests
        exports = [
            {
                'file': f,
                'volume': volume,
                'location': location + '/' + f.name, 
                'overwrite': overwrite
            }
            for f in files[i:i + chunk_size]
        ]
 
        # initiate bulk export of this batch and wait until finished
        responses = api.exports.bulk_submit(exports, copy_only=copy_only)
        while any([is_running(r) for r in responses]):
            time.sleep(10)
            responses = api.exports.bulk_get([r.resource for r in responses])
             
        final_responses.extend(responses)
         
    return final_responses
 
files_to_export = list(
    api.files.query(
        project=src_project,
        names=source_file_names,
        limit=100
    ).all())
 
responses = bulk_export_files(
    files=files_to_export,
    volume=volume,
    location='christian_demo_files',
    copy_only=False
)

Updated 6 months ago

Exporting files to a volume


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.