The StreamFlow framework is a container-native Workflow Management System written in Python 3. It relies on the Common Workflow Language (CWL) standard. Developed and maintained by the Alpha research group at Università di Torino (UniTO), it has been designed around two main principles:
- Allow the execution of tasks in multi-container environments to support the concurrent execution of multiple communicating tasks in a multi-agent ecosystem
- Relax the requirement of a single shared data space to allow hybrid workflow executions on multi-cloud or hybrid cloud/HPC infrastructures.
StreamFlow source code is available on GitHub under the LGPLv3 license. A Python package is downloadable from PyPI, and Docker containers can be found on Docker Hub.
Start learning StreamFlow by reading the documentation or exploring some existing applications that rely on it for large-scale workflow orchestration. If you want to cite it, please refer to this article:
I. Colonnelli, B. Cantalupo, I. Merelli, and M. Aldinucci, “StreamFlow: cross-breeding cloud with HPC,” in IEEE Transactions on Emerging Topics in Computing, vol. 9, iss. 4, p. 1723-1737, 2021. doi: 10.1109/TETC.2020.3019202.