SlurmConnector
The Slurm connector allows offloading execution to High-Performance Computing (HPC) facilities orchestrated by the Slurm queue manager. The HPC facility is supposed to be constantly active, reducing the deployment phase to creating an SSHConnection pointing to its login node.
Interaction with the Slurm scheduler happens through a Bash script with #SBATCH directives. Users can pass the path of a custom script to the connector using the file attribute. This file is interpreted as a Jinja2 template and populated at runtime by the connector.
The unit of binding is the entire HPC facility. In contrast, the scheduling unit is a single job placement in the Slurm queue. Users can limit the maximum number of concurrently placed jobs by setting the maxConcurrentJobs parameter.
properties |
||
hostname (required) |
Hostname of the HPC facility |
|
type |
string |
|
username (required) |
Username needed to connect with the SSH environment |
|
type |
string |
|
checkHostKey |
Perform a strict validation of the host SSH keys (and return exception if key is not recognized as valid) |
|
type |
boolean |
|
default |
True |
|
dataTransferConnection |
Sometimes HPC clusters provide dedicated hostnames for large data transfers, which guarantee a higher efficiency for data movements |
|
type |
string |
|
file |
Path to a file containing a Jinja2 template, describing how the StreamFlow command should be executed in the remote environment |
|
type |
string |
|
maxConcurrentJobs |
Maximum number of jobs concurrently scheduled for execution on the Queue Manager |
|
type |
integer |
|
default |
1 |
|
maxConcurrentSessions |
Maximum number of concurrent session to open for a single SSH client connection |
|
type |
integer |
|
default |
10 |
|
passwordFile |
Path to a file containing the password to use for authentication |
|
type |
string |
|
pollingInterval |
Time interval (in seconds) between consecutive termination checks |
|
type |
integer |
|
default |
5 |
|
sshKey |
Path to the SSH key needed to connect with Slurm environment |
|
type |
string |
|
sshKeyPassphraseFile |
Path to a file containing the passphrase protecting the SSH key |
|
type |
string |
|
transferBufferSize |
Buffer size allocated for local and remote data transfers |
|
type |
integer |
|
default |
16kiB |
|
tunnel |
External SSH connection parameters for tunneling |
|
type |
||