HEPCloud, an Elastic Hybrid HEP Facility using an Intelligent Decision Support System
Authors:
Parag Mhashilkar,
Mine Altunay,
Eileen Berman,
David Dagenhart,
Stuart Fuess,
Burt Holzman,
James Kowalkowski,
Dmitry Litvintsev,
Qiming Lu,
Alexander Moibenko,
Marc Paterno,
Panagiotis Spentzouris,
Steven Timm,
Anthony Tiradani,
Eric Vaandering,
John Hover,
Jose Caballero Bejar
Abstract:
HEPCloud is rapidly becoming the primary system for provisioning compute resources for all Fermilab-affiliated experiments. In order to reliably meet the peak demands of the next generation of High Energy Physics experiments, Fermilab must plan to elastically expand its computational capabilities to cover the forecasted need. Commercial cloud and allocation-based High Performance Computing (HPC) r…
▽ More
HEPCloud is rapidly becoming the primary system for provisioning compute resources for all Fermilab-affiliated experiments. In order to reliably meet the peak demands of the next generation of High Energy Physics experiments, Fermilab must plan to elastically expand its computational capabilities to cover the forecasted need. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and at which scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources spanning multiple cloud providers, multiple HPC centers, and grid computing federations. In this paper, we discuss the goals and architecture of the HEPCloud Facility, the architecture of the IDSS, and our early experience in using the IDSS for automated facility expansion both at Fermi and Brookhaven National Laboratory.
△ Less
Submitted 18 April, 2019;
originally announced April 2019.
Intelligently-automated facilities expansion with the HEPCloud Decision Engine
Authors:
Mine Altunay,
W. David Dagenhart,
Stuart Fuess,
Burt Holzman,
Jim Kowalkowski,
Dmitry Litvintsev,
Qiming Lu,
Parag Mhashilkar,
Alexander Moibenko,
Marc Paterno,
Panagiotis Spentzouris,
Steven Timm,
Anthony Tiradani
Abstract:
The next generation of High Energy Physics experiments are expected to generate exabytes of data---two orders of magnitude greater than the current generation. In order to reliably meet peak demands, facilities must either plan to provision enough resources to cover the forecasted need, or find ways to elastically expand their computational capabilities. Commercial cloud and allocation-based High…
▽ More
The next generation of High Energy Physics experiments are expected to generate exabytes of data---two orders of magnitude greater than the current generation. In order to reliably meet peak demands, facilities must either plan to provision enough resources to cover the forecasted need, or find ways to elastically expand their computational capabilities. Commercial cloud and allocation-based High Performance Computing (HPC) resources both have explicit and implicit costs that must be considered when deciding when to provision these resources, and to choose an appropriate scale. In order to support such provisioning in a manner consistent with organizational business rules and budget constraints, we have developed a modular intelligent decision support system (IDSS) to aid in the automatic provisioning of resources---spanning multiple cloud providers, multiple HPC centers, and grid computing federations.
△ Less
Submitted 11 June, 2018; v1 submitted 8 June, 2018;
originally announced June 2018.
