Skip to main content

Pipelines

Pipelines orchestrate the execution of BatchX images. Their execution results in the creation of a set of jobs, that we refer to as "workflow".

In BatchX pipelines can run client-side (outside of BatchX), or as BatchX jobs.

Client-side pipelines​

BatchX jobs can be orchestrated client-side by using our APIs or CLI commands, by using the workflow management system of choice of the user.

Workflow management​

Client-side pipelines are responsible of managing the workflow creation and closing, according to the following life cycle:

  • Call workflow-create command to create a new workflow
  • Perform job orchestration by passing the workflow token to the submit/run __commands
  • If:
    • everything went well: Call workflow-close
    • an error ocurred, call workflow-close passing the error description

BatchX pipelines​

Pipelines can run as BatchX jobs also. In order to do so, the image needs to be declared as a pipeline image (supported by manifest versions onwards 06).

Runtime considerations​

The platform handles different regular images jobs than pipelines jobs. In particular, for pipeline containers:

  • Input files are not downloaded to the local filesystem. Instead, they are kept as remote BatchX filesystem references in the input.json file.
  • An additional set of environment variables is available, aimed at configuring the BatchX API clients or CLI, to perform the job orchestration.BATCHX_TOKEN, BATCHX_ENDPOINT, BATCHX_SECURE_CONNECTION, BATCHX_ENV

Workflow management​

BatchX pipelines are not allowed to manage the workflow by themselves, instead it is managed by the platform.