============= FluxConnector ============= The `Flux Framework `_ connector allows running jobs on a cluster with Flux Framework in a High Performance Computing Context. Although Flux can work in a local testing container or a cloud environment and has a Python SDK, to match the design here, we follow suit and inherit from the :ref:`QueueManagerConnector `. In this way, users can offload jobs to local or remote PBS controllers using the :ref:`stacked locations ` mechanism. The HPC facility is supposed to be constantly active, reducing the deployment phase to deploy the inner connector (e.g., to create an :ref:`SSHConnection ` pointing to an HPC login node). .. warning:: Note that in StreamFlow ``v0.1``, the ``QueueManagerConnector`` directly inherited from the :ref:`SSHConnector ` at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., ``hostname``, ``username``, and ``sshKey``) were defined directly in the ``QueueManagerConnector``. This path is still supported by StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``. Interaction with the Flux scheduler happens through a Bash script with ``#flux`` directives. Users can pass the path of a custom script to the connector using the ``file`` attribute of the :ref:`FluxService ` configuration. This file is interpreted as a `Jinja2 `_ template and populated at runtime by the connector. Alternatively, users can pass PBS options directly from YAML using the other options of a :ref:`FluxService ` object. As an example, suppose to have a Flux template script called ``batch.sh``, with the following content: .. code-block:: bash #!/bin/bash #flux --nodes=1 #flux --queue=queue_name {{streamflow_command}} A PBS deployment configuration which uses the ``batch.sh`` file to spawn jobs can be written as follows: .. code-block:: yaml deployments: flux-example: type: pbs config: services: example: file: batch.sh Alternatively, the same behaviour can be recreated by directly passing options through the YAML configuration, as follows: .. code-block:: yaml deployments: flux-example: type: pbs config: services: example: nodes: 1 queue: queue_name Being passed directly to the ``flux batch`` command line, the YAML options have higher priority than the file-based ones. .. warning:: Note that the ``file`` property in the upper configuration level, i.e., outside a ``service`` definition, is still supported in StreamFlow ``v0.2``, but it is deprecated and will be removed in StreamFlow ``v0.3``. For a quick demo or tutorial, see our `example workflow `_. .. jsonschema:: ../../../streamflow/deployment/connector/schemas/flux.json :lift_description: true :lift_definitions: true :auto_reference: true :auto_target: true