-
HTC Scientific Computing in a Distributed Cloud Environment
Authors:
R. Sobie,
A. Agarwal,
I. Gable,
C. Leavett-Brown,
M. Paterson,
R. Taylor,
A. Charbonneau,
R. Impey,
W. Podiama
Abstract:
This paper describes the use of a distributed cloud computing system for high-throughput computing (HTC) scientific applications. The distributed cloud computing system is composed of a number of separate Infrastructure-as-a-Service (IaaS) clouds that are utilized in a unified infrastructure. The distributed cloud has been in production-quality operation for two years with approximately 500,000 co…
▽ More
This paper describes the use of a distributed cloud computing system for high-throughput computing (HTC) scientific applications. The distributed cloud computing system is composed of a number of separate Infrastructure-as-a-Service (IaaS) clouds that are utilized in a unified infrastructure. The distributed cloud has been in production-quality operation for two years with approximately 500,000 completed jobs where a typical workload has 500 simultaneous embarrassingly-parallel jobs that run for approximately 12 hours. We review the design and implementation of the system which is based on pre-existing components and a number of custom components. We discuss the operation of the system, and describe our plans for the expansion to more sites and increased computing capacity.
△ Less
Submitted 7 February, 2013;
originally announced February 2013.
-
Data Intensive High Energy Physics Analysis in a Distributed Cloud
Authors:
R. J. Sobie,
A. Agarwal,
M. Anderson,
P. Armstrong,
K. Fransham,
I. Gable,
D. Harris,
C. Leavett-Brown,
M. Paterson,
D. Penfold-Brown,
M. Vliet,
A. Charbonneau,
R. Impey,
W. Podaima
Abstract:
We show that distributed Infrastructure-as-a-Service (IaaS) compute clouds can be effectively used for the analysis of high energy physics data. We have designed a distributed cloud system that works with any application using large input data sets requiring a high throughput computing environment. The system uses IaaS-enabled science and commercial clusters in Canada and the United States. We des…
▽ More
We show that distributed Infrastructure-as-a-Service (IaaS) compute clouds can be effectively used for the analysis of high energy physics data. We have designed a distributed cloud system that works with any application using large input data sets requiring a high throughput computing environment. The system uses IaaS-enabled science and commercial clusters in Canada and the United States. We describe the process in which a user prepares an analysis virtual machine (VM) and submits batch jobs to a central scheduler. The system boots the user-specific VM on one of the IaaS clouds, runs the jobs and returns the output to the user. The user application accesses a central database for calibration data during the execution of the application. Similarly, the data is located in a central location and streamed by the running application. The system can easily run one hundred simultaneous jobs in an efficient manner and should scale to many hundreds and possibly thousands of user jobs.
△ Less
Submitted 1 January, 2011;
originally announced January 2011.
-
Cloud Scheduler: a resource manager for distributed compute clouds
Authors:
P. Armstrong,
A. Agarwal,
A. Bishop,
A. Charbonneau,
R. Desmarais,
K. Fransham,
N. Hill,
I. Gable,
S. Gaudet,
S. Goliath,
R. Impey,
C. Leavett-Brown,
J. Ouellete,
M. Paterson,
C. Pritchet,
D. Penfold-Brown,
W. Podaima,
D. Schade,
R. J. Sobie
Abstract:
The availability of Infrastructure-as-a-Service (IaaS) computing clouds gives researchers access to a large set of new resources for running complex scientific applications. However, exploiting cloud resources for large numbers of jobs requires significant effort and expertise. In order to make it simple and transparent for researchers to deploy their applications, we have developed a virtual mach…
▽ More
The availability of Infrastructure-as-a-Service (IaaS) computing clouds gives researchers access to a large set of new resources for running complex scientific applications. However, exploiting cloud resources for large numbers of jobs requires significant effort and expertise. In order to make it simple and transparent for researchers to deploy their applications, we have developed a virtual machine resource manager (Cloud Scheduler) for distributed compute clouds. Cloud Scheduler boots and manages the user-customized virtual machines in response to a user's job submission. We describe the motivation and design of the Cloud Scheduler and present results on its use on both science and commercial clouds.
△ Less
Submitted 30 June, 2010;
originally announced July 2010.
