The BatchX filesystem

All BatchX environments provide a scalable filesystem to store the input data and results of your jobs.

Content-addressable storage

A BatchX file-system is implemented as a content-addressable storage, a way to store immutable information so it can be retrieved based on its content, not on its location. This offers the benefits of avoiding increasing storage quota when duplicated content is added or aliased.

Blobs

We call "blobs" to the unique data that multiple files pointers can share. Each blob contributes to the filesystem total size only once, no matter how many file pointers they share. File pointers (also called "aliases"), are created when a file is copied within an environment. File pointers are decreased when a file is deleted. A blob is eliminated, and its data removed, when it has no file pointers associated.

Integrity checks

All files transmitted to/from the BatchX filesystem are checked for integrity and automatically resent if a digest mismatch is found.