Stacked locations

StreamFlow supports the concept of stacked locations, adhering to the separation of concerns principle. This allows the user to describe complex execution environments, e.g., a Singularity container launched by a Slurm queue manager called through an SSH connection.

Users can define stacked locations using the wraps property in the StreamFlow file. For example, consider a remote Slurm queue manager that can be contacted by connecting to the login node of an HPC facility using SSH. This is a typical configuration for HPC systems. Then a user can write:

deployments:
  ssh-hpc:
    type: ssh
    config:
      ...
  slurm-hpc:
    type: slurm
    config:
      ...
    wraps: ssh-hpc

Warning

Note that in StreamFlow v0.1, the queue manager connectors (Slurm and PBS) are inherited from the SSHConnector at the implementation level. Consequently, all the properties needed to open an SSH connection to the HPC login node (e.g., hostname, username, and sshKey) were defined directly in the config section of the queue manager deployment. This path is still supported by StreamFlow v0.2, but it is deprecated and will be removed in StreamFlow v0.3.

Note that not all deployment types can wrap other locations. Indeed, only connectors extending the ConnectorWrapper interface support the wraps directive. Specifying the wraps directive on a container type that does not support it will result in an error during StreamFlow initialization. Conversely, if no explicit wraps directive is specified for a ConnectorWrapper, it wraps the LocalConnector.

The wraps directive only supports wrapping a single inner location. However, a single location can be wrapped by multiple deployment definitions. The DeploymentManager component must guarantee the correct deployment and undeployment order for stacked locations.

It is also possible to wrap a single service instead of generically wrapping the whole deployment. This feature can be helpful when dealing with complex deployments that describe entire microservices architectures. To do that, it is necessary to specify the target service name in the StreamFlow file as follows:

deployments:
 slurm-compose:
   type: docker-compose
   config:
     ...
 slurm:
   type: slurm
   config:
     ...
   wraps:
     deployment: slurm-compose
     service: controller