SlurmConnector

The Slurm connector allows offloading execution to High-Performance Computing (HPC) facilities orchestrated by the Slurm queue manager. The HPC facility is supposed to be constantly active, reducing the deployment phase to creating an SSHConnection pointing to its login node.

Interaction with the Slurm scheduler happens through a Bash script with #SBATCH directives. Users can pass the path of a custom script to the connector using the file attribute. This file is interpreted as a Jinja2 template and populated at runtime by the connector.

The unit of binding is the entire HPC facility. In contrast, the scheduling unit is a single job placement in the Slurm queue. Users can limit the maximum number of concurrently placed jobs by setting the maxConcurrentJobs parameter.

properties

hostname (required)

Hostname of the HPC facility

type

string

username (required)

Username needed to connect with the SSH environment

type

string

checkHostKey

Perform a strict validation of the host SSH keys (and return exception if key is not recognized as valid)

type

boolean

default

True

dataTransferConnection

Sometimes HPC clusters provide dedicated hostnames for large data transfers, which guarantee a higher efficiency for data movements

type

string

SSHConnection

file

Path to a file containing a Jinja2 template, describing how the StreamFlow command should be executed in the remote environment

type

string

maxConcurrentJobs

Maximum number of jobs concurrently scheduled for execution on the Queue Manager

type

integer

default

1

maxConcurrentSessions

Maximum number of concurrent session to open for a single SSH client connection

type

integer

default

10

passwordFile

Path to a file containing the password to use for authentication

type

string

pollingInterval

Time interval (in seconds) between consecutive termination checks

type

integer

default

5

sshKey

Path to the SSH key needed to connect with Slurm environment

type

string

sshKeyPassphraseFile

Path to a file containing the passphrase protecting the SSH key

type

string

transferBufferSize

Buffer size allocated for local and remote data transfers

type

integer

default

16kiB

tunnel

External SSH connection parameters for tunneling

type

SSHConnection