S3 file access

All tools accepting files as input, can get them from the BatchX environment or from external S3 buckets. To reference an S3 file in an input message use strings under this form:

s3://<s3-bucket-name>/<s3-file1-key>

Configuration

In order to reference S3 files, you first need to register the bucket into your environment though the add-s3 CLI command:

bx add-s3 -b=<bucket> [-c=<bucket-credentials>]

This command makes BatchX validate the bucket existence, obtain the region it is located in, and store its access credentials for future usage.

Example

Configure bucket

First, configure sra-pub-sars-cov2 public bucket, hosting NCBI data for novel corona virus:

bx add-s3 -b sra-pub-src-2

Clone image

Clone batchx@world-count to count the number of words on a given text file

bx clone batchx@tutorial/word-count

Process an S3 bucket file

Submit the job passing an S3 URL as the input file value:

bx submit -v=1 -m=1000 batchx@tutorial/word-count:2.0.1 '{ \
  "textFile": "s3://sra-pub-sars-cov2/README.txt"
}'