The Seven Bridges Knowledge Center

The Seven Bridges Platform is a simple solution for doing bioinformatics at industrial scale. But sometimes, everyone needs a little help.

Get Started

Importing files from a volume

The Seven Bridges API allows you to import files from a volume in bulk rather than one by one. Using the bulk API feature, you can import up to 100 files per call.

Not optimized for rate limit

Importing individual files requires two API calls for each file: one to find the file by name, and another one to import it. We recommend using the bulk API call instead.

for f in files_to_import:
     
    imported_file = api.imports.submit_import(
        volume=volume,
        project=dest_project,
        location='christian_demo_files/' + f,
        overwrite=True
    )

Optimized for rate limit

Using the bulk API feature, you will first query all files that need to be imported and then use one API to import up to 100 files.

def bulk_import_files(file_names, volume, location, project, overwrite=True, chunk_size=100):
    "Imports list of files from volume in bulk"
 
    def is_running(response):
        if not response.valid or response.resource.error:
            raise Exception(
                '\n'.join([
                    str(response.resource.error),
                    response.resource.error.message,
                    response.resource.error.more_info
                ]))
        return response.resource.state not in ["COMPLETED", "FAILED", "ABORTED"]
     
    final_responses = []
     
    # import files in batches of 100 each
    for i in range(0, len(file_names), chunk_size):
         
        # setup list of dictionary with import requests
        imports = [
            {
                'volume': volume,
                'location': location + '/' + fn,
                'project': project,
                'name' : fn, 
                'overwrite': True
            }
            for fn in file_names[i:i + chunk_size]
        ]
 
        # initiate bulk import of batch and wait until finished
        responses = api.imports.bulk_submit(imports)
        while any([is_running(r) for r in responses]):
            time.sleep(10)
            responses = api.imports.bulk_get([r.resource for r in responses])
             
        final_responses.extend(responses)
         
    return final_responses
 
 
responses = bulk_import_files(
    file_names=files_to_import,
    volume=volume,
    location="christian_demo_files",
    project=dest_project
)

Updated about a year ago

Importing files from a volume


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.