Journal article
Conference paper
Book chapter
Thesis
2026
-
[J9]
, “QSplit: A workflow-oriented hybrid quantum-classical optimization framework,” Euro-Par 2026: Parallel Processing - 32nd European Conference on Parallel and Distributed Processing, Pisa, Italy, August 24-28, 2026, Proceedings, Part II, vol. 16782, pp. 195–209, 2026.
Abstract
Current quantum processors are too limited to solve large combinatorial optimization problems directly, motivating hybrid executions that combine quantum processors with classical HPC resources. This paper presents QSplit, a workflow-oriented hybrid quantum-classical framework for distributed optimization. QSplit targets problems expressed in the Quadratic Unconstrained Binary Optimization form, decomposes large instances into smaller sub-instances, dispatches them to heterogeneous classical or quantum backends, and reconstructs a global solution within a portable and reproducible workflow. The orchestration layer of the QSplit workflow is managed by StreamFlow WMS, enabling configurable execution across cloud, HPC, and remote quantum infrastructures. We evaluate QSplit on Knapsack and Max-Cut instances to study how problem structure affects decomposition quality. Results show that QSplit achieves competitive behavior on sparse instances, where decomposition preserves enough structure for effective aggregation, while dense instances expose the limits of purely structural splitting due to the loss of global correlations. These findings suggest that QSplit provides a framework for studying hybrid HPC-quantum optimization pipelines and their current scalability boundaries.DOIBibTeX
@inproceedings{2026:euro-par:qsplit, author = {Mario Bifulco and Francesco Medina and Doriana Medi\'{c} and Luca Roversi and Marco Aldinucci}, editor = {Massimo Torquati and Gabriele Mencagli and Valeria Cardellini and Alessia Antelmi and Doriana Medi\'{c}}, title = {QSplit: {A} Workflow-Oriented Hybrid Quantum-Classical Optimization Framework}, booktitle = {Euro-Par 2026: Parallel Processing - 32nd European Conference on Parallel and Distributed Processing, Pisa, Italy, August 24-28, 2026, Proceedings, Part {II}}, series = {Lecture Notes in Computer Science}, volume = {16782}, pages = {195--209}, publisher = {Springer}, year = {2026}, doi = {10.1007/978-3-032-35251-4_14}, } -
[J8]
, “Three ways to share a QPU: Scheduling strategies for hybrid quantum-HPC applications,” Future Generation Computer Systems, vol. 185, pp. 108699, 2026.
Abstract
As quantum computing (QC) technologies mature, their integration into established high-performance computing (HPC) infrastructures is becoming a central objective for next-generation computing systems. However, unlocking the potential of hybrid platforms for computationally demanding workloads remains challenging. The mismatch between quantum and classical programming models, the limited maturity of quantum software stacks, and the scarcity of quantum processing units (QPUs) above all, necessitate scheduling strategies that go beyond standard HPC mechanisms to manage such heterogeneous and constrained resources. To address this issue, we investigate three distinct methodologies for HPC-QC resource scheduling: time-based multiplexing, dynamic resource management, and workflow decomposition. Experimental validation on production HPC clusters and real quantum hardware demonstrates the effectiveness of these approaches under different workload scenarios. Malleability and workflow strategies significantly optimise classical resource utilisation, reducing consumption by up to 45.7% and 64% respectively, proving to be best fitted for hybrid jobs where quantum and classical workloads are evenly balanced. Conversely, time-multiplexing enhances QPU utilisation and reduces execution time at the cluster level, making it the optimal strategy for the opposite context, which is characterised by high classical-quantum workload imbalances. These findings underscore the practical viability of tailored scheduling strategies for hybrid HPC-QC environments and highlight their complementarity in building efficient, scalable software stacks for next-generation quantum-accelerated facilities.DOIBibTeX
@article{2026:fgcs:qpu, author = {Marco Cipollini and Simone Rizzo and Sergio Iserte and Paolo Viviani and Giacomo Vitali and Matteo Barbieri and Gabriella Bettonte and Elisabetta Boella and Fulvio Ganz and Roberto Rocco and Orazio Spina and Antonio J. Pe\~{n}a and Petter Sand\r{a}s and Iacopo Colonnelli and Alberto Scionti and Chiara Vercellino and Emanuele Dri and Jonathan Frassineti and Sara Marzella and Andrea Muratori and Daniele Ottaviani and Olivier Terzo and Bartolomeo Montrucchio and Daniele Gregori}, title = {Three ways to share a QPU: Scheduling strategies for hybrid Quantum-HPC applications}, journal = {Future Generation Computer Systems}, volume = {185}, pages = {108699}, year = 2026, issn = {0167-739X}, doi = {10.1016/j.future.2026.108699}, } -
[J7]
, “Hybrid streaming workflows for scalable visualization in astronomy and cosmology,” SN Computer Science, vol. 7, pp. 527, 2026.
Abstract
Modern astrophysics and cosmology generate petabyte-scale datasets that demand advanced tools for scalable visualization and analysis. The Visualization Interface for the Virtual Observatory (VisIVO) provides multi-dimensional data exploration, but its integration with heterogeneous computing infrastructures and data-intensive workflows has remained challenging. Here we present a portable and reproducible approach that combines VisIVO with the StreamFlow workflow engine and the CAPIO middleware. StreamFlow enables hybrid execution across cloud-HPC environments, while CAPIO injects transparent I/O streaming into legacy file-based pipelines without modifying application code. Together, these technologies accelerate high–performance visualization, improve reproducibility, and reduce I/O bottlenecks. We demonstrate our approach on large-scale cosmological simulations produced with ChaNGa and GADGET-based codes, including studies of massive neutrinos in the DEMNUni suite. Our results show that streaming-enhanced workflows deliver up to 50% performance gains and enable interactive, scalable visualization across distributed infrastructures, paving the way for exascale-ready scientific discovery.DOIBibTeX
@article{2026:sn:visivo, author = {Sciacca, Eva and Tuccari, Nicola and Santimaria, Marco Edoardo and Cesare, Valentina and Vitello, Fabio and Mulone, Alberto and Medi{\'{c}}, Doriana and Colonnelli, Iacopo}, title = {Hybrid Streaming Workflows for Scalable Visualization in Astronomy and Cosmology}, journal = {SN Computer Science}, year = {2026}, month = jun, volume = 7, number = 5, pages = 527, issn = {2661-8907}, doi = {10.1007/s42979-026-05076-4}, } -
[J6]
, “A formal framework for fault tolerance in hybrid scientific workflows,” Future Generation Computer Systems, vol. 176, pp. 108188, 2026.
Abstract
In large-scale distributed systems, failures are routine events whose occurrences increase with the number of computational tasks and execution locations. The advantage of representing an application as a workflow is the possibility of exploiting Workflow Management System (WMS) features such as portability, scalability, and, crucially, reliability. Among these, reliability is essential for ensuring robust execution in dynamic and failure-prone environments. In recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. Consequently, the number of possible points of failure during the execution increased, creating a need for sophisticated fault tolerance mechanisms capable of addressing the specific requirements of hybrid systems. This work introduces a formal framework for a fault tolerance mechanism in hybrid workflows, enabling failure recovery through a rollback approach. The framework is rigorously defined by adapting and extending an existing workflow semantics tailored for hybrid execution. Our method leverages provenance data from workflow execution up to the point of failure, and creates a recovery workflow that spans multiple infrastructures. The rollback approach provides a robust and reliable strategy to ensure resilience against step failures and potential data loss. We then implement this mechanism in the StreamFlow WMS, and evaluate it using two case studies: the 1000 Genomes workflow and a synthetic workflow featuring iterative patterns. Experiments showcase the conceptual validity of our approach and assess the overhead introduced by the mechanism, including data availability checks.DOIBibTeX
@article{25:fgcs:ft, title = {A formal framework for fault tolerance in hybrid scientific workflows}, author = {Alberto Mulone and Doriana Medi\'{c} and Iacopo Colonnelli and Marco Aldinucci}, year = {2026}, journal = {{F}uture {G}eneration {C}omputer {S}ystems}, volume = {176}, pages = {108188}, doi = {10.1016/j.future.2025.108188}, issn = {0167-739X}, } -
[J5]
, “A terminology for scientific workflow systems,” Future Generation Computer Systems, vol. 174, pp. 107974, 2026.
Abstract
The term “scientific workflow” has evolved over the last two decades to encompass a broad range of compositions of interdependent compute tasks and data movements. It has also become an umbrella term for processing in modern scientific applications. Today, many scientific applications can be considered as workflows made of multiple dependent steps, and hundreds of workflow systems have been developed to manage and run these scientific workflows. However, no turnkey solution has emerged from the field to address the diversity of scientific processes and the infrastructure on which they are supposed to be implemented. Instead, new research problems requiring the execution of scientific workflows with some novel feature often lead to the development of an entirely new workflow system. A direct consequence of this situation is that many existing workflow management systems (WMSs) share some salient features, offer similar functionalities, and can manage the same categories of workflows but at the same time also have some distinct capabilities that can be important for specific applications. This situation makes researchers who develop workflows face the complex question of selecting a WMS. This selection can be driven by technical considerations, to find the system that is the most appropriate for their application and for the computing and storage resources available to them, or other factors such as reputation, adoption, strong community support, or long-term sustainability. To address this problem, a group of WMS developers and practitioners joined their efforts to produce a community-based terminology of WMSs. This paper summarizes their findings and introduces this new terminology to characterize WMSs. This terminology is composed of fives axes: workflow structure and characteristics, composition, orchestration, data management, and metadata capture. Each axis comprises several concepts that capture the prominent features of WMSs. Based on this terminology, this paper also presents a classification of 23 existing WMSs according to the proposed axes and terms.DOIBibTeX
@article{26:fgcs:terminology, title = {A terminology for scientific workflow systems}, author = {Suter, Fr\'{e}d\'{e}ric and Coleman, Tain\~{a} and Altinta\c{s}, \.{I}lkay and Badia, Rosa M. and Balis, Bartosz and Chard, Kyle and Colonnelli, Iacopo and Deelman, Ewa and Di Tommaso, Paolo and Fahringer, Thomas and Goble, Carole and Jha, Shantenu and Katz, Daniel S. and K\"{o}ster, Johannes and Leser, Ulf and Mehta, Kshitij and Oliver, Hilary and Peterson, J.-Luc and Pizzi, Giovanni and Pottier, Lo\"{i}c and Sirvent, Ra\"{u}l and Suchyta, Eric and Thain, Douglas and R. Wilkinson, Sean and Wozniak, Justin M. and Ferreira da Silva, Rafael}, year = 2026, journal = {Future Generation Computer Systems}, volume = {174}, pages = {107974}, doi = {10.1016/j.future.2025.107974}, issn = {0167-739X}, }
2025
-
[C14]
, “Towards RISC-V-based HPC: The italian pathfinding activities in the DARE-SGA1 project,” in 28th Euromicro Conference on Digital System Design, DSD 2025, Salerno, Italy: IEEE, 2025, pp. 360–367.
Abstract
The European Union’s efforts towards technological sovereignty in High-Performance Computing are driving research and development of RISC-V-based supercomputers. The DARE SGA1 project, in particular, aims to develop chips designed and owned by Europeans. This paper introduces the Italian contribution to DARE SGA1 regarding pathfinding activities toward future RISC-V-based accelerator designs, reliability improvements, system software, and AI and Quantum Chemistry applications.DOIBibTeX
@inproceedings{2025:dsd:dare, title = {Towards {RISC-V}-based {HPC:} The Italian Pathfinding Activities in the {DARE-SGA1} Project}, author = {Agosta, Giovanni and Aldinucci, Marco and Bartolini, Andrea and Bellentani, Laura and Biagioni, Andrea and Cesarini, Daniele and Chiarini, Carlotta and Colonnelli, Iacopo and Delugas, Pietro and Denisov, Lev and Frezza, Ottorino and Grangetto, Marco and Lo Cicero, Francesca and Lonardo, Alessandro and Martinelli, Michele and Maslov, Andrea and Olivieri, Mauro and Perticaroli, Pierpaolo and Pontisso, Luca and Rossi, Cristian and Rossi, Davide and Saponara, Sergio and Sciarappa, Antonio and Simula, Francesco and Sonza Reorda, Matteo and Torquati, Massimo and Vicini, Piero}, year = {2025}, booktitle = {28th Euromicro Conference on Digital System Design, {DSD} 2025}, location = {Salerno, Italy}, publisher = {{IEEE}}, pages = {360--367}, doi = {10.1109/DSD67783.2025.00057}, mon = sep, } -
[C13]
, “Dynamic solutions for hybrid quantum-HPC resource allocation,” in 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Albuquerque, NM, USA: IEEE, Sep. 2025, pp. 34–40.
Abstract
The integration of quantum computers within classical High-Performance Computing (HPC) infrastructures is receiving increasing attention, with the former expected to serve as accelerators for specific computational tasks. However, combining HPC and quantum computers presents significant technical challenges, including resource allocation. This paper presents a novel malleability-based approach, alongside a workflow-based strategy, to optimize resource utilization in hybrid HPC-quantum workloads. With both these approaches, we can release classical resources when computations are offloaded to the quantum computer and reallocate them once quantum processing is complete. Our experiments with a hybrid HPC-quantum use case show the benefits of dynamic allocation, highlighting the potential of those solutions.DOIBibTeX
@inproceedings{2025:qce:hybrid, title = {Dynamic Solutions for Hybrid Quantum-HPC Resource Allocation}, author = {Rocco, Roberto and Rizzo, Simone and Barbieri, Matteo and Bettonte, Gabriella and Boella, Elisabetta and Ganz, Fulvio and Iserte, Sergio and Pe{\~{n}}a, Antonio J. and Sand{\aa}s, Petter and Scionti, Alberto and Terzo, Olivier and Vercellino, Chiara and Vitali, Giacomo and Viviani, Paolo and Frassineti, Jonathan and Marzella, Sara and Ottaviani, Daniele and Colonnelli, Iacopo and Gregori, Daniele}, year = {2025}, month = sep, booktitle = {2025 {IEEE} International Conference on Quantum Computing and Engineering (QCE)}, location = {Albuquerque, NM, USA}, publisher = {{IEEE}}, pages = {34--40}, doi = {10.1109/QCE65121.2025.10289}, } -
[C12]
, “EuroHPC SPACE CoE: Redesigning scalable parallel astrophysical codes for exascale. Invited paper,” in Proceedings of the 22nd ACM International Conference on Computing Frontiers: Workshops and Special Sessions, CF 2025, Cagliari, Italy, May 28-30, 2025, Cagliari, Italy: ACM, F. Palumbo, A. Tumeo, A. L. Varbanescu, and Y. Simmhan, Eds., 2025, pp. 177–184.
Abstract
High Performance Computing (HPC) based simulations are crucial in Astrophysics & Cosmology (A&C), helping scientists investigate and understand complex astrophysical phenomena. Taking advantage of exascale computing capabilities is essential for these efforts. However, the unprecedented architectural complexity of exascale systems impacts legacy codes. The SPACE Centre of Excellence (CoE) aims to re-engineer key astrophysical codes to tackle new computational challenges by adopting innovative programming paradigms and software (SW) solutions. SPACE brings together scientists, code developers, HPC experts, hardware (HW) manufacturers, and SW developers. This collaboration enhances exascale A&C applications, promoting the use of exascale and post-exascale computing capabilities. Additionally, SPACE addresses high-performance data analysis for the massive data outputs from exascale simulations and modern observations, using machine learning (ML) and visualisation tools. The project facilitates application deployment across platforms by focusing on code repositories and data sharing, integrating European astrophysical communities around exascale computing with standardised SW and data protocols.DOIBibTeX
@inproceedings{25:cf:space, title = {{EuroHPC} {SPACE} {CoE}: Redesigning Scalable Parallel Astrophysical Codes for Exascale. Invited Paper}, author = {Shukla, Nitin and Romeo, Alessandro and Caravita, Caterina and Riha, Lubomir and Vysocky, Ondrej and Strakos, Petr and Jaros, Milan and Barbosa, Jo{\~{a}}o and Vavr{\'{\i}}k, Radim and Mignone, Andrea and Rossazza, Marco and Truzzi, Stefano and Berta, Vittoria and Colonnelli, Iacopo and Medi\'{c}, Doriana and Boella, Elisabetta and Gregori, Daniele and Sciacca, Eva and Tornatore, Luca and Taffoni, Giuliano and Deka, Pranab J. and Bacchini, Fabio and Wilhelm, Rostislav{-}Paul and Doulis, Georgios and Pierre, Khalil and Rezzolla, Luciano and Colman, Tine and Commer{\c{c}}on, Beno{\^{\i}}t and Bouizi, Othman and Kuhn, Matthieu and Raffin, Erwan and Sergent, Marc and Wissing, Robert and Marin, Guillermo and Dolag, Klaus and Karademir, Geray S. and Perna, Gino and Zanotti, Marisa and Trujillo{-}Gomez, Sebastian}, year = {2025}, booktitle = {Proceedings of the 22nd {ACM} International Conference on Computing Frontiers: Workshops and Special Sessions, {CF} 2025, Cagliari, Italy, May 28-30, 2025}, location = {Cagliari, Italy}, publisher = {{ACM}}, pages = {177--184}, doi = {10.1145/3706594.3728892}, editor = {Francesca Palumbo and Antonino Tumeo and Ana Lucia Varbanescu and Yogesh Simmhan}, } -
[C11]
, “The cloud-HPC infrastructure for hazard mapping and vulnerability monitoring (HaMMon),” in 2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP): IEEE, 2025, pp. 309–316.
Abstract
The HaMMon project is the outcome of an industrial partnership that includes many Italian research institutions and private companies. It is led by UnipolSai and Leitha, and funded by the ICSC, the Italian National Research Center for High Performance Computing, Big Data and Quantum Computing.The ambition of HaMMon is to build a flexible and scalable platform to analyze the hydrogeological and atmospheric balance of the Italian territory. The project aims to expand the current knowledge in hazard mapping, monitoring, and forecasting from an industrial perspective by leveraging innovative technologies and the interdisciplinary activities carried out by the ICSC.In this work, we present the cloud-HPC infrastructure deployed in the High-Performance Computing for Artificial Intelligence (HPC4AI) green data center of the University of Turin which supports the testing and development of HaMMon’s applications and services. We describe the current activities and preliminary results related to the integration of Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.DOIBibTeX
@inproceedings{25:pdp:hammon, title = {The Cloud-HPC infrastructure for Hazard Mapping and vulnerability Monitoring (HaMMon)}, author = {Mauro Imbrosciano and Eva Sciacca and Fabio Vitello and Leonardo Pelonero and Francesco Franchina and Ugo Becciani and Iacopo Colonnelli and Doriana Medi\'{c}}, year = {2025}, booktitle = {2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)}, publisher = {IEEE}, pages = {309--316}, doi = {10.1109/PDP66500.2025.00050}, issn = {2377-5750}, } -
[C10]
, “High performance visualization for astrophysics and cosmology,” in 2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP): IEEE, 2025, pp. 443–450.
Abstract
Modern Astrophysics and Cosmology (A&C) projects produce immense data volumes, necessitating advanced software tools for data access, storage, and analysis. Visualization Interface for the Virtual Observatory (VisIVO) is one such tool enabling multi-dimensional data analysis and knowledge discovery across complex astrophysical datasets. Leveraging containerization and virtualization, VisIVO has been deployed on various distributed computing platforms. Additionally, Blender, an open-source 3D suite, provides robust tools for rendering and processing volumetric data, making it suitable for visualizing complex datasets. At the SPACE Center of Excellence these tools are being adapted for high-performance visualization of cosmological simulations performed with GADGET and ChaNGa on pre-exascale systems. However, implementing high-performance visualization on diverse HPC platforms presents several challenges, including hardware and software compatibility, data management, scalability, performance portability, and efficient resource allocation. This paper outlines strategies to integrate VisIVO with workflow frameworks and streaming platforms to address these challenges. Workflow frameworks enhance portability, scheduling, and reproducibility of visualization workflows on pre-exascale systems used in A&C simulations. We also discuss the use of streaming platforms to enable concurrent (i.e. in-situ) analysis and visualization of simulations, reducing the need to store full simulation data by leveraging distributed databases that stream the output data in real time. Lastly, we present an adaptation of Blender to handle large-scale particle-based astrophysical data, offering high-quality visualization with interactive exploration capabilities.DOIBibTeX
@inproceedings{25:pdp:visivo, title = {High Performance Visualization for Astrophysics and Cosmology}, author = {Tuccari, Nicola and Sciacca, Eva and Vitello, Fabio and Colonnelli, Iacopo and Becerra, Yolanda and Cintero, Enric Sosa and Marin, Guillermo and Jaros, Milan and Riha, Lubomir and Strakos, Petr and Trujillo-Gomez, Sebastian and Tramontana, Emiliano and Wissing, Robert}, year = {2025}, booktitle = {2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP)}, publisher = {IEEE}, pages = {443--450}, doi = {10.1109/PDP66500.2025.00069}, issn = {2377-5750}, } -
[C9]
, “Towards a european HPC/AI ecosystem: A community-driven report,” in Procedia Computer Science, Amsterdam, Netherlands: Elsevier, 2025, pp. 140–149.
Abstract
The rapid advancements in AI and Machine Learning necessitate a robust computational infrastructure to support cutting-edge research and industrial applications. From the academic and industrial AI community perspective, voiced in the recent ELISE project, the European AI platform is recommended to center around the EuroHPC growing ecosystem. It should be user-driven, easily accessible, powerful, and compliant with European regulations. AI-optimized and dedicated supercomputers for the European AI community are also coming, in addition to upgrading partitions of existing EuroHPC systems to ’AI enabled’ stage. Related calls have been initiated in September 2024. Further, conventional EuroHPC systems are suggested to be extended with quantum computing, edge AI, and neuromorphic computing to cater to AI models deployed on network edge devices and sustainability in the long run. The challenges are presented in three case studies, ranging from training Transformers on HPC to LLMs trained federally across three different Euro HPC systems to recent results on hybrid classical-quantum application. This paper concludes with case studies results-informed next steps believed to benefit AI practitioners and the broader AI community.DOIBibTeX
@article{24:eurohpc:hpcai, title = {Towards a European {HPC}/{AI} ecosystem: a community-driven report}, author = {Taborsky, Petr and Colonnelli, Iacopo and Kurowski, Krzysztof and Sarma, Rakesh and Pontoppidan, Niels Henrik and Jans\'{i}k, Branislav and Detlefsen, Nicki Skafte and Egholm Pedersen, Jens and Larsen, Rasmus and Hansen, Lars Kai}, year = {2025}, journal = {Procedia Computer Science}, booktitle = {Proceedings of the Second EuroHPC user day}, publisher = {Elsevier}, address = {Amsterdam, Netherlands}, volume = {255}, pages = {140--149}, doi = {10.1016/j.procs.2025.02.269}, issn = {1877-0509}, }
2024
-
[J4]
, “Recording provenance of workflow runs with RO-Crate,” PLoS ONE, vol. 19, pp. 1–35, 2024.
Abstract
Recording the provenance of scientific computation results is key to the support of traceability, reproducibility and quality assessment of data products. Several data models have been explored to address this need, providing representations of workflow plans and their executions as well as means of packaging the resulting information for archiving and sharing. However, existing approaches tend to lack interoperable adoption across workflow management systems. In this work we present Workflow Run RO-Crate, an extension of RO-Crate (Research Object Crate) and Schema.org to capture the provenance of the execution of computational workflows at different levels of granularity and bundle together all their associated objects (inputs, outputs, code, etc.). The model is supported by a diverse, open community that runs regular meetings, discussing development, maintenance and adoption aspects. Workflow Run RO-Crate is already implemented by several workflow management systems, allowing interoperable comparisons between workflow runs from heterogeneous systems. We describe the model, its alignment to standards such as W3C PROV, and its implementation in six workflow systems. Finally, we illustrate the application of Workflow Run RO-Crate in two use cases of machine learning in the digital image analysis domain.DOIBibTeX
@article{24:plosone:leo, title = {Recording provenance of workflow runs with {RO-Crate}}, author = {Simone Leo and Michael R. Crusoe and Laura Rodr\'{i}guez-Navas and Raül Sirvent and Alexander Kanitz and Paul De Geest and Rudolf Wittner and Luca Pireddu and Daniel Garijo and Jos\'{e} M. Fern\'{a}ndez and Iacopo Colonnelli and Matej Gallo and Tazro Ohta and Hirotaka Suetake and Salvador Capella-Gutierrez and Renske Wit and Bruno P. Kinoshita and Stian Soiland-Reyes}, doi = {10.1371/journal.pone.0309210}, year = {2024}, date = {2024-09-01}, journal = {PLoS ONE}, volume = {19}, number = {9}, pages = {1--35}, publisher = {Public Library of Science}, } -
[J3]
, “Cross-facility federated learning,” Procedia Computer Science, vol. 240, pp. 3–12, 2024.
Abstract
In a decade, AI frontier research transitioned from the researcher’s workstation to thousands of high-end hardware-accelerated compute nodes. This rapid evolution shows no signs of slowing down in the foreseeable future. While top cloud providers may be able to keep pace with this growth rate, obtaining and efficiently exploiting computing resources at that scale is a daunting challenge for universities and SMEs. This work introduces the Cross-Facility Federated Learning (XFFL) framework to bridge this compute divide, extending the opportunity to efficiently exploit multiple independent data centres for extreme-scale deep learning tasks to data scientists and domain experts. XFFL relies on hybrid workflow abstractions to decouple tasks from environment-specific technicalities, reducing complexity and enhancing reusability. In addition, Federated Learning (FL) algorithms eliminate the need to move large amounts of data between different facilities, reducing time-to-solution and preserving data privacy. The XFFL approach is empirically evaluated by training a full LLaMAv2 7B instance on two facilities of the EuroHPC JU, showing how the increased computing power completely compensates for the additional overhead introduced by two data centres.DOIBibTeX
@article{24:procs:colonnelli, title = {Cross-Facility Federated Learning}, author = {Iacopo Colonnelli and Robert Birke and Giulio Malenza and Gianluca Mittone and Alberto Mulone and Jeroen Galjaard and Lydia Y. Chen and Sanzio Bassini and Gabriella Scipione and Jan Martinovi\v{c} and Vit Vondr\'{a}k and Marco Aldinucci}, doi = {10.1016/j.procs.2024.07.003}, issn = {1877-0509}, year = {2024}, booktitle = {Proceedings of the First EuroHPC user day}, journal = {Procedia Computer Science}, volume = {240}, pages = {3--12}, publisher = {Elsevier}, address = {Bruxelles, Belgium}, } -
[C8]
, “A fault tolerance mechanism for hybrid scientific workflows,” in 1st workshop about High-Performance e-Science (HiPES), Madrid, Spain, 2024.
Abstract
In large distributed systems, failures are a daily event occurring frequently, especially with growing numbers of computation tasks and locations on which they are deployed. The advantage of representing an application as a workflow is possibility to utilize the Workflow Management Systems which are reliable systems guaranteeing the correct execution of the application and providing the features such as portability, scalability, and fault tolerance. Over recent years, the emergence of hybrid workflows has posed new and intriguing challenges by increasing the possibility of distributing computations involving heterogeneous and independent environments. As a consequence, the number of possible points of failure in the execution augmented, creating different important challenges interesting to study.DOIBibTeX
@inproceedings{24:hipes:mulone, title = {A Fault Tolerance mechanism for Hybrid Scientific Workflows}, author = {Alberto Mulone and Doriana Medi\'{c} and Marco Aldinucci}, year = {2024}, booktitle = {1st workshop about High-Performance e-Science (HiPES)}, location = {Madrid, Spain}, doi = {10.1007/978-3-031-90203-1_13}, }
2023
-
[C7]
, “A systematic mapping study of italian research on workflows,” in Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023, Denver, CO, USA: ACM, 2023, pp. 2065–2076.
Abstract
An entire ecosystem of methodologies and tools revolves around scientific workflow management. They cover crucial non-functional requirements that standard workflow models fail to target, such as interactive execution, energy efficiency, performance portability, Big Data management, and intelligent orchestration in the Computing Continuum. Characterizing and monitoring this ecosystem is crucial to develop an informed view of current and future research directions. This work conducts a systematic mapping study of the Italian workflow research community, collecting and analyzing 25 tools and 10 applications from several scientific domains in the context of the "National Research Centre for HPC, Big Data, and Quantum Computing" (ICSC). The study aims to outline the main current research directions and determine how they address the critical needs of modern scientific applications. The findings highlight a variegated research ecosystem of tools, with a prominent interest in advanced workflow orchestration and still immature but promising efforts toward energy efficiency.DOIBibTeX
@inproceedings{23:scw:aldinucci, title = {A Systematic Mapping Study of Italian Research on Workflows}, author = {Marco Aldinucci and Elena Maria Baralis and Valeria Cardellini and Iacopo Colonnelli and Marco Danelutto and Sergio Decherchi and Giuseppe Di Modica and Luca Ferrucci and Marco Gribaudo and Francesco Iannone and Marco Lapegna and Doriana\'{c} and Giuseppa Muscianisi and Francesca Righetti and Eva Sciacca and Nicola Tonellotto and Mauro Tortonesi and Paolo Trunfio and Tullio Vardanega}, doi = {10.1145/3624062.3624285}, year = {2023}, booktitle = {Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023}, pages = {2065--2076}, publisher = {ACM}, address = {Denver, CO, USA}, } -
[C6]
, “Porting the variant calling pipeline for NGS data in cloud-HPC environment,” in 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, Torino, Italy: IEEE, H. Shahriar, Y. Teranishi, A. Cuzzocrea, M. Sharmin, D. Towey, A. K. M. J. A. Majumder, H. Kashiwazaki, J. Yang, M. Takemoto, N. Sakib, R. Banno, and S. I. Ahamed, Eds., 2023, pp. 1858–1863.
Abstract
In recent years we have understood the importance of analyzing and sequencing human genetic variation. A relevant aspect that emerged from the Covid-19 pandemic was the need to obtain results very quickly; this involved using High-Performance Computing (HPC) environments to execute the Next Generation Sequencing (NGS) pipeline. However, HPC is not always the most suitable environment for the entire execution of a pipeline, especially when it involves many heterogeneous tools. The ability to execute parts of the pipeline on different environments can lead to higher performance but also cheaper executions. This work shows the design and optimization process that led us to a state-of-the-art Variant Calling hybrid workflow based on the StreamFlow Workflow Management System (WfMS). We also compare StreamFlow with Snakemake, an established WfMS targeting HPC facilities, observing comparable performance on single environments and satisfactory improvements with a hybrid cloud-HPC configuration.DOIBibTeX
@inproceedings{23:compsac:mulone, title = {Porting the Variant Calling Pipeline for {NGS} data in cloud-{HPC} environment}, author = {Alberto Mulone and Sherine Awad and Davide Chiarugi and Marco Aldinucci}, editor = {Hossain Shahriar and Yuuichi Teranishi and Alfredo Cuzzocrea and Moushumi Sharmin and Dave Towey and A. K. M. Jahangir Alam Majumder and Hiroki Kashiwazaki and Ji-Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Sheikh Iqbal Ahamed}, doi = {10.1109/COMPSAC57700.2023.00288}, year = {2023}, booktitle = {47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023}, pages = {1858--1863}, publisher = {IEEE}, address = {Torino, Italy}, } -
[C5]
, “Federated learning meets HPC and cloud,” in Astrophysics and Space Science Proceedings, Catania, Italy: Springer, F. Bufano, S. Riggi, E. Sciacca, and F. Schillirò, Eds., 2023, pp. 193–199.
Abstract
HPC and AI are fated to meet for several reasons. This article will discuss some of them and argue why this will happen through the set of methods and technologies that underpin cloud computing. As a paradigmatic example, we present a new federated learning system that collaboratively trains a deep learning model in different supercomputing centers. The system is based on the StreamFlow workflow manager designed for hybrid cloud-HPC infrastructures.DOIBibTeX
@inproceedings{23:ml4astro:colonnelli, title = {Federated Learning meets {HPC} and cloud}, author = {Iacopo Colonnelli and Bruno Casella and Gianluca Mittone and Yasir Arfat and Barbara Cantalupo and Roberto Esposito and Alberto Riccardo Martinelli and Doriana Medi\'{c} and Marco Aldinucci}, editor = {Filomena Bufano and Simone Riggi and Eva Sciacca and Francesco Schillir\`{o}}, doi = {10.1007/978-3-031-34167-0_39}, isbn = {978-3-031-34167-0}, year = {2023}, booktitle = {Astrophysics and Space Science Proceedings}, volume = {60}, pages = {193--199}, publisher = {Springer}, address = {Catania, Italy}, } -
[C4]
, “Workflow models for heterogeneous distributed systems,” in Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023), Naples, Italy: CEUR-WS.org, N. Bena, B. D. Martino, A. Maratea, A. Sperduti, E. D. Nardo, A. Ciaramella, R. Montella, and C. A. Ardagna, Eds., 2023.
Abstract
This article introduces a novel hybrid workflow abstraction that injects topology awareness directly into the definition of a distributed workflow model. In particular, the article briefly discusses the advantages brought by this approach to the design and orchestration of large-scale data-oriented workflows, the current level of support from state-of-the-art workflow systems, and some future research directions.BibTeX
@inproceedings{23:itadata:colonnelli, title = {Workflow Models for Heterogeneous Distributed Systems}, author = {Iacopo Colonnelli}, editor = {Nicola Bena and Beniamino Di Martino and Antonio Maratea and Alessandro Sperduti and Emanuel Di Nardo and Angelo Ciaramella and Raffaele Montella and Claudio A. Ardagna}, year = {2023}, booktitle = {Proceedings of the 2nd Italian Conference on Big Data and Data Science (ITADATA 2023)}, volume = {3606}, publisher = {CEUR-WS.org}, series = {CEUR Workshop Proceedings}, address = {Naples, Italy}, } -
[B5]
, “Bringing cell subpopulation discovery on a cloud-HPC using rCASC and StreamFlow,” in Single Cell Transcriptomics: Methods and Protocols, New York, NY: Springer US, R. A. Calogero and V. Benes, Eds., 2023, pp. 337–345.
Abstract
The idea behind novel single-cell RNA sequencing (scRNA-seq) pipelines is to isolate single cells through microfluidic approaches and generate sequencing libraries in which the transcripts are tagged to track their cell of origin. Modern scRNA-seq platforms are capable of analyzing up to many thousands of cells in each run. Then, combined with massive high-throughput sequencing producing billions of reads, scRNA-seq allows the assessment of fundamental biological properties of cell populations and biological systems at unprecedented resolution.DOIBibTeX
@inbook{23:springer:contaldo, title = {Bringing Cell Subpopulation Discovery on a Cloud-{HPC} Using {rCASC} and {StreamFlow}}, author = {Sandro Gepiro Contaldo and Luca Alessandri and Iacopo Colonnelli and Marco Beccuti and Marco Aldinucci}, editor = {Raffaele Adolfo Calogero and Vladimir Benes}, doi = {10.1007/978-1-0716-2756-3_17}, isbn = {978-1-0716-2756-3}, year = {2023}, booktitle = {Single Cell Transcriptomics: Methods and Protocols}, pages = {337--345}, publisher = {Springer US}, address = {New York, NY}, }
2022
-
[J2]
, “Distributed workflows with Jupyter,” Future Generation Computer Systems, vol. 128, pp. 282–298, 2022.
Abstract
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.DOIBibTeX
@article{22:fgcs:colonnelli, title = {Distributed workflows with {Jupyter}}, author = {Iacopo Colonnelli and Marco Aldinucci and Barbara Cantalupo and Luca Padovani and Sergio Rabellino and Concetto Spampinato and Roberto Morelli and Rosario Di Carlo and Nicol\`{o} Magini and Carlo Cavazzoni}, doi = {10.1016/j.future.2021.10.007}, issn = {0167-739X}, year = {2022}, journal = {Future Generation Computer Systems}, volume = {128}, pages = {282--298}, } -
[B4]
, “The DeepHealth toolkit: A key european free and open-source software for deep learning and computer vision ready to exploit heterogeneous HPC and cloud architectures,” in Technologies and Applications for Big Data Value, Cham: Springer International Publishing, E. Curry, S. Auer, A. J. Berre, A. Metzger, M. S. Perez, and S. Zillner, Eds., 2022, pp. 183–202.
Abstract
At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.DOIBibTeX
@incollection{22:springer:aldinucci, title = {The {DeepHealth} Toolkit: {A} Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous {HPC} and Cloud Architectures}, author = {Marco Aldinucci and David Atienza and Federico Bolelli and M\'{o}nica Caballero and Iacopo Colonnelli and Jos\'{e} Flich and Jon Ander G\'{o}mez and David Gonz\'{a}lez and Costantino Grana and Marco Grangetto and Simone Leo and Pedro L\'{o}pez and Dana Oniga and Roberto Paredes and Luca Pireddu and Eduardo Quiñones and Tatiana Silva and Enzo Tartaglione and Marina Zapater}, editor = {Edward Curry and S\"{o}ren Auer and Arne J. Berre and Andreas Metzger and Maria S. Perez and Sonja Zillner}, doi = {10.1007/978-3-030-78307-5_9}, isbn = {978-3-030-78307-5}, year = {2022}, booktitle = {Technologies and Applications for Big Data Value}, pages = {183--202}, publisher = {Springer International Publishing}, address = {Cham}, chapter = {9}, } -
[B3]
, “The DeepHealth HPC infrastructure: Leveraging heterogenous HPC and cloud computing infrastructures for IA-based medical solutions,” in HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, Boca Raton, Florida: CRC Press, O. Terzo and J. Martinovič, Eds., 2022, pp. 191–216.
Abstract
This chapter presents the DeepHealth HPC toolkit for an efficient execution of deep learning (DL) medical application into HPC and cloud-computing infrastructures, featuring many-core, GPU, and FPGA acceleration devices. The toolkit offers to the European Computer Vision Library and the European Distributed Deep Learning Library (EDDL), developed in the DeepHealth project as well, the mechanisms to distribute and parallelize DL operations on HPC and cloud infrastructures in a fully transparent way. The toolkit implements workflow managers used to orchestrate HPC workloads for an efficient parallelization of EDDL training operations on HPC and cloud infrastructures, and includes the parallel programming models for an efficient execution EDDL inference and training operations on many-core, GPUs and FPGAs acceleration devices.DOIBibTeX
@incollection{22:crcpress:quinones, title = {The {DeepHealth} {HPC} Infrastructure: {Leveraging} Heterogenous {HPC} and Cloud Computing Infrastructures for {IA}-based Medical Solutions}, author = {Eduardo Qui\~{n}ones and Jesus Perales and Jorge Ejarque and Asaf Badouh and Santiago Marco and Fabrice Auzanneau and François Galea and David Gonz\'{a}lez and Jos\'{e} Ram\'{o}n Herv\'{a}s and Tatiana Silva and Iacopo Colonnelli and Barbara Cantalupo and Marco Aldinucci and Enzo Tartaglione and Rafael Tornero and José Flich and Jose Maria Martinez and David Rodriguez and Izan Catal\'{a}n and Jorge Garcia and Carles Hern\'{a}ndez}, editor = {Olivier Terzo and Jan Martinovi\v{c}}, doi = {10.1201/9781003176664}, isbn = {978-1-0320-0984-1}, year = {2022}, booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision}, pages = {191--216}, publisher = {CRC Press}, address = {Boca Raton, Florida}, chapter = {10}, } -
[B2]
, “Toward the convergence of high-performance computing, cloud, and big data domains,” in HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, Boca Raton, Florida: CRC Press, O. Terzo and J. Martinovič, Eds., 2022, pp. 1–16.
Abstract
Convergence between big data, high-performance computing, and the cloud is the key driving factor for sustainable economic growth in the future. Technological advances in many fields are determined by competence to gain precise information from the large amounts of data collected, which in turn requires powerful computing resources. This chapter provides an overview on the evolution of the three fields and four different points of view on their convergence provided by the CYBELE, DeepHealth, Evolve, and LEXIS projects funded by the European Union under the Horizon 2020 Programme.DOIBibTeX
@incollection{22:crcpress:golasowski, title = {Toward the Convergence of High-Performance Computing, Cloud, and Big Data Domains}, author = {Martin Golasowski and Jan Martinovič and Marc Levrier and Stephan Hachinger and Sophia Karagiorgou and Aikaterini Papapostolou and Spiros Mouzakitis and Ioannis Tsapelas and Monica Caballero and Marco Aldinucci and Jon Ander Gómez and Antony Chazapis and Jean-Thomas Acquaviva}, editor = {Olivier Terzo and Jan Martinovi\v{c}}, doi = {10.1201/9781003176664}, isbn = {978-1-0320-0984-1}, year = {2022}, booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision}, pages = {1--16}, publisher = {CRC Press}, address = {Boca Raton, Florida}, chapter = {1}, } -
[B1]
, “Applications of AI and HPC in the health domain,” in HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision, Boca Raton, Florida: CRC Press, O. Terzo and J. Martinovič, Eds., 2022, pp. 217–239.
Abstract
This chapter presents the applications of artificial intelligence (AI) and high-computing performance (HPC) in the health domain, illustrated by the description of five of the use cases that are developed in the DeepHealth project. In the context of the European Commission supporting the use of AI and HPC in the health sector, DeepHealth Project is helping health experts process large quantities of images, putting at their disposal DeepLearning and computer vision techniques, combined in the DeepHealth toolkit and HPC infrastructures. The DeepHealth toolkit is tested and validated through 15 use cases, each of them representing a biomedical application. The most promising use cases are described in the chapter, which concludes with the value proposition and the benefits that DeepHealth toolkit offers to future end users.DOIBibTeX
@incollection{22:crcpress:oniga, title = {Applications of {AI} and {HPC} in the Health Domain}, author = {Dana Oniga and Barbara Cantalupo and Enzo Tartaglione and Daniele Perlo and Marco Grangetto and Marco Aldinucci and Federico Bolelli and Federico Pollastri and Michele Cancilla and Laura Canalini and Costantino Grana and Cristina Mu\~{n}oz Alcalde and Franco Alberto Cardillo and Monica Florea}, editor = {Olivier Terzo and Jan Martinovi\v{c}}, doi = {10.1201/9781003176664}, isbn = {978-1-0320-0984-1}, year = {2022}, booktitle = {HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision}, pages = {217--239}, publisher = {CRC Press}, address = {Boca Raton, Florida}, chapter = {11}, }
2021
-
[J1]
, “StreamFlow: Cross-breeding cloud with HPC,” IEEE Transactions on Emerging Topics in Computing, vol. 9, pp. 1723–1737, 2021.
Abstract
The Cloud-HPC continuum is attracting considerable interest from the academic and industrial communities, thanks to its ability to provide seamless access to a wide range of computing resources with heterogeneous characteristics. However, the orchestration of workloads across such heterogeneous infrastructures is still an open challenge, due to the lack of standard interfaces and the intrinsic complexity of managing multiple sites. In this work, we present StreamFlow, a workflow management system designed to support the execution of complex workflows on hybrid Cloud/HPC infrastructures. StreamFlow relies on the Common Workflow Language (CWL) standard for the description of workflow steps and their dependencies, and on a novel declarative language for the description of the execution environment. The StreamFlow framework is evaluated on a real-world bioinformatics workflow, showing its ability to seamlessly execute the same workflow on top of different infrastructures, with comparable performance.DOIBibTeX
@article{21:tetc:colonnelli, title = {{StreamFlow}: cross-breeding cloud with {HPC}}, author = {Iacopo Colonnelli and Barbara Cantalupo and Iacopo Merelli and Marco Aldinucci}, doi = {10.1109/TETC.2020.3019202}, year = {2021}, journal = {IEEE Transactions on Emerging Topics in Computing}, volume = {9}, number = {4}, pages = {1723--1737}, } -
[C3]
, “TEXTAROSSA: Towards EXtreme scale Technologies and Accelerators for euROhpc hw/Sw Supercomputing Applications for exascale,” in Proc. of the 24th Euromicro Conference on Digital System Design (DSD), Palermo, Italy: IEEE, 2021.
Abstract
To achieve high performance and high energy efficiency on near-future exascale computing systems, three key technology gaps needs to be bridged. These gaps include: energy efficiency and thermal control; extreme computation efficiency via HW acceleration and new arithmetics; methods and tools for seamless integration of reconfigurable accelerators in heterogeneous HPC multi-node platforms. TEXTAROSSA aims at tackling this gap through a co-design approach to heterogeneous HPC solutions, supported by the integration and extension of HW and SW IPs, programming models and tools derived from European research.DOIBibTeX
@inproceedings{21:dsd:agosta, title = {{TEXTAROSSA}: {Towards} {EX}treme scale {T}echnologies and {A}ccelerators for eu{RO}hpc hw/{S}w {S}upercomputing {A}pplications for exascale}, author = {Giovanni Agosta and William Fornaciari and Andrea Galimberti and Giuseppe Massari and Federico Reghenzani and Federico Terraneo and Davide Zoni and Carlo Brandolese and Massimo Celino and Francesco Iannone and Paolo Palazzari and Giuseppe Zummo and Massimo Bernaschi and Pasqua D'Ambra and Sergio Saponara and Marco Danelutto and Massimo Torquati and Marco Aldinucci and Yasir Arfat and Barbara Cantalupo and Iacopo Colonnelli and Roberto Esposito and Alberto Riccardo Martinelli and Gianluca Mittone and Olivier Beaumont and Berenger Bramas and Lionel Eyraud-Dubois and Brice Goglin and Abdou Guermouche and Raymond Namyst and Samuel Thibault and Antonio Filgueras and Miquel Vidal and Carlos Alvarez and Xavier Martorell and Ariel Oleksiak and Michal Kulczewski and Alessandro Lonardo and Piero Vicini and Francesco Lo Cicero and Francesco Simula and Andrea Biagioni and Paolo Cretaro and Ottorino Frezza and Pier Stanislao Paolucci and Matteo Turisini and Francesco Giacomini and Tommaso Boccali and Simone Montangero and Roberto Ammendola}, doi = {10.1109/DSD53832.2021.00051}, year = {2021}, booktitle = {Proc. of the 24th Euromicro Conference on Digital System Design (DSD)}, publisher = {IEEE}, address = {Palermo, Italy}, } -
[C2]
, “Bringing AI pipelines onto cloud-HPC: Setting a baseline for accuracy of COVID-19 diagnosis,” in ENEA CRESCO in the fight against COVID-19: ENEA, F. Iannone, Ed., 2021.
Abstract
HPC is an enabling platform for AI. The introduction of AI workloads in the HPC applications basket has non-trivial consequences both on the way of designing AI applications and on the way of providing HPC computing. This is the leitmotif of the convergence between HPC and AI. The formalized definition of AI pipelines is one of the milestones of HPC-AI convergence. If well conducted, it allows, on the one hand, to obtain portable and scalable applications. On the other hand, it is crucial for the reproducibility of scientific pipelines. In this work, we advocate the StreamFlow Workflow Management System as a crucial ingredient to define a parametric pipeline, called "CLAIRE COVID-19 Universal Pipeline", which is able to explore the optimization space of methods to classify COVID-19 lung lesions from CT scans, compare them for accuracy, and therefore set a performance baseline. The universal pipeline automatizes the training of many different Deep Neural Networks (DNNs) and many different hyperparameters. It, therefore, requires a massive computing power, which is found in traditional HPC infrastructure thanks to the portability-by-design of pipelines designed with StreamFlow. Using the universal pipeline, we identified a DNN reaching over 90DOIBibTeX
@inproceedings{21:enea:colonnelli, title = {Bringing {AI} pipelines onto cloud-{HPC}: setting a baseline for accuracy of {COVID-19} diagnosis}, author = {Iacopo Colonnelli and Barbara Cantalupo and Concetto Spampinato and Matteo Pennisi and Marco Aldinucci}, editor = {Francesco Iannone}, doi = {10.5281/zenodo.5151511}, year = {2021}, booktitle = {ENEA CRESCO in the fight against COVID-19}, publisher = {ENEA}, } -
[C1]
, “HPC application cloudification: The StreamFlow toolkit,” in 12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021): Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2021.
Abstract
Modern scientific workflows are often composed of heterogeneous tasks, requiring different computational resources and software environments. These tasks are typically supported by different sites, such as Cloud and HPC facilities, each one providing a different interface and requiring a different deployment strategy. In this work, we present the StreamFlow toolkit, a workflow management system designed to support the execution of complex workflows on hybrid Cloud/HPC infrastructures. We describe the architecture, the main features, and the usage scenarios of StreamFlow, showing how it can be used to seamlessly execute workflows on top of heterogeneous resources.DOIBibTeX
@inproceedings{21:oasics:colonnelli, title = {{HPC} Application Cloudification: {The} {StreamFlow} Toolkit}, author = {Iacopo Colonnelli and Barbara Cantalupo and Roberto Esposito and Matteo Pennisi and Concetto Spampinato and Marco Aldinucci}, doi = {10.4230/OASIcs.PARMA-DITAM.2021.5}, year = {2021}, booktitle = {12th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and 10th Workshop on Design Tools and Architectures for Multicore Embedded Computing Platforms (PARMA-DITAM 2021)}, publisher = {Schloss Dagstuhl -- Leibniz-Zentrum f\"{u}r Informatik}, series = {Open Access Series in Informatics (OASIcs)}, }