=============
FluxConnector
=============
The `Flux Framework `_ connector allows running jobs on a cluster with Flux Framework in a High Performance Computing Context. Although Flux can work in a local testing container or a cloud environment and has a Python SDK, to match the design here, we follow suit and inherit from the :ref:`QueueManagerConnector `. In this way, users can offload jobs to local or remote PBS controllers using the :ref:`stacked locations ` mechanism. The HPC facility is supposed to be constantly active, reducing the deployment phase to deploy the inner connector (e.g., to create an :ref:`SSHConnection ` pointing to an HPC login node).
.. warning::
Note that in StreamFlow ``v0.1``, the ``QueueManagerConnector`` directly inherited from the :ref:`SSHConnector ` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``QueueManagerConnector``. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
Interaction with the Flux scheduler happens through a Bash script with ``#flux`` directives. Users can pass the path of a custom script to the connector using the ``file`` attribute of the :ref:`FluxService ` configuration. This file is interpreted as a `Jinja2 `_ template and populated at runtime by the connector. Alternatively, users can pass PBS options directly from YAML using the other options of a :ref:`FluxService ` object.
As an example, suppose to have a Flux template script called ``batch.sh``, with the following content:
.. code-block:: bash
#!/bin/bash
#flux --nodes=1
#flux --queue=queue_name
{{streamflow_command}}
A PBS deployment configuration which uses the ``batch.sh`` file to spawn jobs can be written as follows:
.. code-block:: yaml
deployments:
flux-example:
type: pbs
config:
services:
example:
file: batch.sh
Alternatively, the same behaviour can be recreated by directly passing options through the YAML configuration, as follows:
.. code-block:: yaml
deployments:
flux-example:
type: pbs
config:
services:
example:
nodes: 1
queue: queue_name
Being passed directly to the ``flux batch`` command line, the YAML options have higher priority than the file-based ones.
.. warning::
Note that the ``file`` property in the upper configuration level, i.e., outside a ``service`` definition, is still supported in StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``.
For a quick demo or tutorial, see our `example workflow `_.
.. jsonschema:: ../../../streamflow/deployment/connector/schemas/flux.json
:lift_description: true
:lift_definitions: true
:auto_reference: true
:auto_target: true