-
HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation
Authors:
Burt Holzman,
Lothar A. T. Bauerdick,
Brian Bockelman,
Dave Dykstra,
Ian Fisk,
Stuart Fuess,
Gabriele Garzoglio,
Maria Girone,
Oliver Gutsche,
Dirk Hufnagel,
Hyunwoo Kim,
Robert Kennedy,
Nicolo Magini,
David Mason,
Panagiotis Spentzouris,
Anthony Tiradani,
Steve Timm,
Eric W. Vaandering
Abstract:
Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly de…
▽ More
Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing nterest among the cloud providers to demonstrate the capability to perform large-scale scientific computing. In this paper, we discuss results from the CMS experiment using the Fermilab HEPCloud facility, which utilized both local Fermilab resources and virtual machines in the Amazon Web Services Elastic Compute Cloud. We discuss the planning, technical challenges, and lessons learned involved in performing physics workflows on a large-scale set of virtualized resources. In addition, we will discuss the economics and operational efficiencies when executing workflows both in the cloud and on dedicated resources.
△ Less
Submitted 29 September, 2017;
originally announced October 2017.
-
Management of Grid Jobs and Information within SAMGrid
Authors:
A. Baranovski,
G. Garzoglio,
A. Kreymer,
L. Lueking,
S. Stonjek,
I. Terekhov,
F. Wuerthwein,
A. Roy,
P. Mhashikar,
V. Murthi,
T. Tannenbaum,
R. Walker,
F. Ratnikov,
T. Rockwell
Abstract:
We describe some of the key aspects of the SAMGrid system, used by the D0 and CDF experiments at Fermilab. Having sustained success of the data handling part of SAMGrid, we have developed new services for job and information services. Our job management is rooted in \CondorG and uses enhancements that are general applicability for HEP grids. Our information system is based on a uniform framework…
▽ More
We describe some of the key aspects of the SAMGrid system, used by the D0 and CDF experiments at Fermilab. Having sustained success of the data handling part of SAMGrid, we have developed new services for job and information services. Our job management is rooted in \CondorG and uses enhancements that are general applicability for HEP grids. Our information system is based on a uniform framework for configuration management based on XML data representation and processing.
△ Less
Submitted 8 July, 2003; v1 submitted 3 July, 2003;
originally announced July 2003.
-
Adapting SAM for CDF
Authors:
D. Bonham,
G. Garzoglio,
R. Herber,
J. Kowalkowski,
D. Litvintsev,
L. Lueking,
M. Paterno,
D. Petravick,
L. Piccoli,
R. Pordes,
N. Stanfield,
I. Terekhov,
J. Trumbo,
J. Tseng,
S. Veseli,
M. Votava,
V. White,
T. Huffman,
S. Stonjek,
K. Waltkins,
P. Crosby,
D. Waters,
R. St. Denis
Abstract:
The CDF and D0 experiments probe the high-energy frontier and as they do so have accumulated hundreds of Terabytes of data on the way to petabytes of data over the next two years. The experiments have made a commitment to use the developing Grid based on the SAM system to handle these data. The D0 SAM has been extended for use in CDF as common patterns of design emerged to meet the similar requi…
▽ More
The CDF and D0 experiments probe the high-energy frontier and as they do so have accumulated hundreds of Terabytes of data on the way to petabytes of data over the next two years. The experiments have made a commitment to use the developing Grid based on the SAM system to handle these data. The D0 SAM has been extended for use in CDF as common patterns of design emerged to meet the similar requirements of these experiments. The process by which the merger was achieved is explained with particular emphasis on lessons learned concerning the database design patterns plus realization of the use cases.
△ Less
Submitted 18 June, 2003;
originally announced June 2003.
