Install

You can install StreamFlow as a Python package with pip, run it in a Docker container or deploy it on Kubernetes with Helm.

Pip

The StreamFlow module is available on PyPI, so you can install it using the following command:

pip install streamflow

Please note that StreamFlow requires python >= 3.8 to be installed on the system. Then you can execute your workflows through the StreamFlow CLI:

streamflow /path/to/streamflow.yml

Docker

StreamFlow Docker images are available on Docker Hub. To download the latest StreamFlow image, you can use the following command:

docker pull alphaunito/streamflow:latest

The command below gives an example of how to execute a StreamFlow workflow in a Docker container:

docker run -d \
--mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
--mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
--mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
alphaunito/streamflow \
/streamflow/project/streamflow.yml

Note

A StreamFlow project, containing a streamflow.yml file and all the other relevant dependencies (e.g. a CWL description of the workflow steps and a Helm description of the execution environment) need to be mounted as a volume inside the container, for example in the /streamflow/project folder.

By default, workflow outputs will be stored in the /streamflow/results folder. Therefore, it is necessary to mount such location as a volume in order to persist the results.

StreamFlow will save all its temporary files inside the /tmp/streamflow location. For debugging purposes, or in order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume.

Warning

All the container-based connectors (i.e., DockerConnector, DockerComposeConnector and SingularityConnector) are not supported from inside a Docker container, as running nested containers is a non-trivial task.

Kubernetes

It is also possible to execute the StreamFlow container as a Job in Kubernetes, with the same characteristics and restrictions discussed for the Docker case. A Helm template of a StreamfFlow Job can be found here.

In this case, the StreamFlow HelmConnector is able to deploy Helm models directly on the parent cluster, relying on ServiceAccount credentials.

Warning

In case RBAC is active on the Kubernetes cluster, a proper RoleBinding must be attached to the ServiceAccount object, in order to give StreamFlow the permissions to manage deployments of pods and executions of tasks.