-
Benchmarking FedAvg and FedCurv for Image Classification Tasks
Authors:
Bruno Casella,
Roberto Esposito,
Carlo Cavazzoni,
Marco Aldinucci
Abstract:
Classic Machine Learning techniques require training on data available in a single data lake. However, aggregating data from different owners is not always convenient for different reasons, including security, privacy and secrecy. Data carry a value that might vanish when shared with others; the ability to avoid sharing the data enables industrial applications where security and privacy are of par…
▽ More
Classic Machine Learning techniques require training on data available in a single data lake. However, aggregating data from different owners is not always convenient for different reasons, including security, privacy and secrecy. Data carry a value that might vanish when shared with others; the ability to avoid sharing the data enables industrial applications where security and privacy are of paramount importance, making it possible to train global models by implementing only local policies which can be run independently and even on air-gapped data centres. Federated Learning (FL) is a distributed machine learning approach which has emerged as an effective way to address privacy concerns by only sharing local AI models while keeping the data decentralized. Two critical challenges of Federated Learning are managing the heterogeneous systems in the same federated network and dealing with real data, which are often not independently and identically distributed (non-IID) among the clients. In this paper, we focus on the second problem, i.e., the problem of statistical heterogeneity of the data in the same federated network. In this setting, local models might be strayed far from the local optimum of the complete dataset, thus possibly hindering the convergence of the federated model. Several Federated Learning algorithms, such as FedAvg, FedProx and Federated Curvature (FedCurv), aiming at tackling the non-IID setting, have already been proposed. This work provides an empirical assessment of the behaviour of FedAvg and FedCurv in common non-IID scenarios. Results show that the number of epochs per round is an important hyper-parameter that, when tuned appropriately, can lead to significant performance gains while reducing the communication cost. As a side product of this work, we release the non-IID version of the datasets we used so to facilitate further comparisons from the FL community.
△ Less
Submitted 31 March, 2023;
originally announced March 2023.
-
Experimenting with Normalization Layers in Federated Learning on non-IID scenarios
Authors:
Bruno Casella,
Roberto Esposito,
Antonio Sciarappa,
Carlo Cavazzoni,
Marco Aldinucci
Abstract:
Training Deep Learning (DL) models require large, high-quality datasets, often assembled with data from different institutions. Federated Learning (FL) has been emerging as a method for privacy-preserving pooling of datasets employing collaborative training from different institutions by iteratively globally aggregating locally trained models. One critical performance challenge of FL is operating…
▽ More
Training Deep Learning (DL) models require large, high-quality datasets, often assembled with data from different institutions. Federated Learning (FL) has been emerging as a method for privacy-preserving pooling of datasets employing collaborative training from different institutions by iteratively globally aggregating locally trained models. One critical performance challenge of FL is operating on datasets not independently and identically distributed (non-IID) among the federation participants. Even though this fragility cannot be eliminated, it can be debunked by a suitable optimization of two hyper-parameters: layer normalization methods and collaboration frequency selection. In this work, we benchmark five different normalization layers for training Neural Networks (NNs), two families of non-IID data skew, and two datasets. Results show that Batch Normalization, widely employed for centralized DL, is not the best choice for FL, whereas Group and Layer Normalization consistently outperform Batch Normalization. Similarly, frequent model aggregation decreases convergence speed and mode quality.
△ Less
Submitted 19 March, 2023;
originally announced March 2023.
-
EXSCALATE: An extreme-scale in-silico virtual screening platform to evaluate 1 trillion compounds in 60 hours on 81 PFLOPS supercomputers
Authors:
Davide Gadioli,
Emanuele Vitali,
Federico Ficarelli,
Chiara Latini,
Candida Manelfi,
Carmine Talarico,
Cristina Silvano,
Carlo Cavazzoni,
Gianluca Palermo,
Andrea Rosario Beccari
Abstract:
The social and economic impact of the COVID-19 pandemic demands the reduction of the time required to find a therapeutic cure. In the contest of urgent computing, we re-designed the Exscalate molecular docking platform to benefit from heterogeneous computation nodes and to avoid scaling issues. We deployed the Exscalate platform on two top European supercomputers (CINECA-Marconi100 and ENI-HPC5),…
▽ More
The social and economic impact of the COVID-19 pandemic demands the reduction of the time required to find a therapeutic cure. In the contest of urgent computing, we re-designed the Exscalate molecular docking platform to benefit from heterogeneous computation nodes and to avoid scaling issues. We deployed the Exscalate platform on two top European supercomputers (CINECA-Marconi100 and ENI-HPC5), with a combined computational power of 81 PFLOPS, to evaluate the interaction between 70 billions of small molecules and 15 binding-sites of 12 viral proteins of Sars-Cov2. The experiment lasted 60 hours and overall it performed a trillion of evaluations.
△ Less
Submitted 22 October, 2021;
originally announced October 2021.
-
COUNTDOWN Slack: a Run-time Library to Reduce Energy Footprint in Large-scale MPI Applications
Authors:
Daniele Cesarini,
Andrea Bartolini,
Andrea Borghesi,
Carlo Cavazzoni,
Mathieu Luisier,
Luca Benini
Abstract:
The power consumption of supercomputers is a major challenge for system owners, users, and society. It limits the capacity of system installations, it requires large cooling infrastructures, and it is the cause of a large carbon footprint. Reducing power during application execution without changing the application source code or increasing time-to-completion is highly desirable in real-life high-…
▽ More
The power consumption of supercomputers is a major challenge for system owners, users, and society. It limits the capacity of system installations, it requires large cooling infrastructures, and it is the cause of a large carbon footprint. Reducing power during application execution without changing the application source code or increasing time-to-completion is highly desirable in real-life high-performance computing scenarios. The power management run-time frameworks proposed in the last decade are based on the assumption that the duration of communication and application phases in an MPI application can be predicted and used at run-time to trade-off communication slack with power consumption. In this manuscript, we first show that this assumption is too general and leads to mispredictions, slowing down applications, thereby jeopardizing the claimed benefits. We then propose a new approach based on (i) the separation of communication phases and slack during MPI calls and (ii) a timeout algorithm to cope with the hardware power management latency, which jointly makes it possible to achieve performance-neutral power saving in MPI applications without requiring labor-intensive and risky application source code modifications. We validate our approach in a tier-1 production environment with widely adopted scientific applications. Our approach has a time-to-completion overhead lower than 1%, while it successfully exploits slack in communication phases to achieve an average energy saving of 10%. If we focus on a large-scale application runs, the proposed approach achieves 22% energy saving with an overhead of only 0.4%. With respect to state-of-the-art approaches, COUNTDOWN Slack is the only that always leads to an energy saving with negligible overhead (<3%).
△ Less
Submitted 27 September, 2019;
originally announced September 2019.
-
Tunable Approximations to Control Time-to-Solution in an HPC Molecular Docking Mini-App
Authors:
Davide Gadioli,
Gianluca Palermo,
Stefano Cherubin,
Emanuele Vitali,
Giovanni Agosta,
Candida Manelfi,
Andrea R. Beccari,
Carlo Cavazzoni,
Nico Sanna,
Cristina Silvano
Abstract:
The drug discovery process involves several tasks to be performed in vivo, in vitro and in silico. Molecular docking is a task typically performed in silico. It aims at finding the three-dimensional pose of a given molecule when it interacts with the target protein binding site. This task is often done for virtual screening a huge set of molecules to find the most promising ones, which will be for…
▽ More
The drug discovery process involves several tasks to be performed in vivo, in vitro and in silico. Molecular docking is a task typically performed in silico. It aims at finding the three-dimensional pose of a given molecule when it interacts with the target protein binding site. This task is often done for virtual screening a huge set of molecules to find the most promising ones, which will be forwarded to the later stages of the drug discovery process. Given the huge complexity of the problem, molecular docking cannot be solved by exploring the entire space of the ligand poses. State-of-the-art approaches face the problem by sampling the space of the ligand poses to generate results in a reasonable time budget. In this work, we improve the geometric approach to molecular docking by introducing tunable approximations. In particular, we analyzed and enriched the original implementation with tunable software knobs to explore and control the performance-accuracy tradeoffs. We modeled time-to-solution of the virtual screening task as a function of software knobs, input data features, and available computational resources. Therefore, the application can autotune its configuration according to a user-defined time budget. We used a Mini-App derived by LiGenDock - a state-of-the-art molecular docking application - to validate the proposed approach. We run the enhanced Mini-App on an HPC system by using a very large database of pockets and ligands. The proposed approach exposes a time-to-solution interval spanning more than one order of magnitude with accuracy degradation up to 30%, more in general providing different accuracy levels according to the needs of the virtual screening campaign.
△ Less
Submitted 18 January, 2019;
originally announced January 2019.
-
Exploiting OpenMP & OpenACC to Accelerate a Molecular Docking Mini-App in Heterogeneous HPC Nodes
Authors:
Emanuele Vitali,
Davide Gadioli,
Gianluca Palermo,
Andrea Beccari,
Carlo Cavazzoni,
Cristina Silvano
Abstract:
In drug discovery, molecular docking is the task in charge of estimating the position of a molecule when interacting with the docking site. This task is usually used to perform screening of a large library of molecules, in the early phase of the process. Given the amount of candidate molecules and the complexity of the application, this task is usually performed using High-Performance Computing (H…
▽ More
In drug discovery, molecular docking is the task in charge of estimating the position of a molecule when interacting with the docking site. This task is usually used to perform screening of a large library of molecules, in the early phase of the process. Given the amount of candidate molecules and the complexity of the application, this task is usually performed using High-Performance Computing (HPC) platforms. In modern HPC systems, heterogeneous platforms provide a better throughput with respect to homogeneous platforms. In this work, we ported and optimized a molecular docking application to a heterogeneous system, with one or more GPU accelerators, leveraging a hybrid OpenMP and OpenACC approach. We prove that our approach has a better exploitation of the node compared to pure CPU/GPU data splitting approaches, reaching a throughput improvement up to 36% while considering the same computing node.
△ Less
Submitted 18 January, 2019;
originally announced January 2019.
-
The ANTAREX Domain Specific Language for High Performance Computing
Authors:
Cristina Silvano,
Giovanni Agosta,
Andrea Bartolini,
Andrea R. Beccari,
Luca Benini,
Loïc Besnard,
João Bispo,
Radim Cmar,
João M. P. Cardoso,
Carlo Cavazzoni,
Daniele Cesarini,
Stefano Cherubin,
Federico Ficarelli,
Davide Gadioli,
Martin Golasowski,
Antonio Libri,
Jan Martinovič,
Gianluca Palermo,
Pedro Pinto,
Erven Rohou,
Kateřina Slaninová,
Emanuele Vitali
Abstract:
The ANTAREX project relies on a Domain Specific Language (DSL) based on Aspect Oriented Programming (AOP) concepts to allow applications to enforce extra functional properties such as energy-efficiency and performance and to optimize Quality of Service (QoS) in an adaptive way. The DSL approach allows the definition of energy-efficiency, performance, and adaptivity strategies as well as their enfo…
▽ More
The ANTAREX project relies on a Domain Specific Language (DSL) based on Aspect Oriented Programming (AOP) concepts to allow applications to enforce extra functional properties such as energy-efficiency and performance and to optimize Quality of Service (QoS) in an adaptive way. The DSL approach allows the definition of energy-efficiency, performance, and adaptivity strategies as well as their enforcement at runtime through application autotuning and resource and power management. In this paper, we present an overview of the key outcome of the project, the ANTAREX DSL, and some of its capabilities through a number of examples, including how the DSL is applied in the context of the project use cases.
△ Less
Submitted 18 January, 2019;
originally announced January 2019.
-
COUNTDOWN: a Run-time Library for Performance-Neutral Energy Saving in MPI Applications
Authors:
Daniele Cesarini,
Andrea Bartolini,
Pietro Bonfà,
Carlo Cavazzoni,
Luca Benini
Abstract:
Power and energy consumption is becoming key challenges to deploy the first exascale supercomputer successfully. Large-scale HPC applications waste a significant amount of power in communication and synchronization-related idle times. However, due to the time scale at which communication happens, transitioning in low power states during communication's idle times may introduce unacceptable overhea…
▽ More
Power and energy consumption is becoming key challenges to deploy the first exascale supercomputer successfully. Large-scale HPC applications waste a significant amount of power in communication and synchronization-related idle times. However, due to the time scale at which communication happens, transitioning in low power states during communication's idle times may introduce unacceptable overhead in applications' execution time. In this paper, we present COUNTDOWN, a runtime library, supported by a methodology and analysis tool for identifying and automatically reducing the power consumption of the computing elements during communication and synchronization. COUNTDOWN saves energy without imposing significant time-to-completion increase by lowering CPUs power consumption only during idle times for which power state transition overhead are negligible. This is done transparently to the user, without requiring labor-intensive and error-prone application code modifications, nor requiring recompilation of the application. We test our methodology in a production Tier-0 system. For the NAS benchmarks, COUNTDOWN saves between 6% and 50% energy, with a time-to-solution penalty lower than 5%. In a complete production --- Quantum ESPRESSO --- for a 3.5K cores run, COUNTDOWN saves 22.36% energy, with a performance penalty below 3%. Energy saving increases to 37% with a performance penalty of 6.38%, if the application is executed without communication tuning.
△ Less
Submitted 23 May, 2019; v1 submitted 19 June, 2018;
originally announced June 2018.
