-
State of In Situ Visualization in Simulations: We are fast. But are we inspiring?
Authors:
Axel Huebl,
Arianna Formenti,
Marco Garten,
Jean-Luc Vay
Abstract:
Visualization of dynamic processes in scientific high-performance computing is an immensely data intensive endeavor. Application codes have recently demonstrated scaling to full-size Exascale machines, and generating high-quality data for visualization is consequently on the machine-scale, easily spanning 100s of TBytes of input to generate a single video frame. In situ visualization, the techniqu…
▽ More
Visualization of dynamic processes in scientific high-performance computing is an immensely data intensive endeavor. Application codes have recently demonstrated scaling to full-size Exascale machines, and generating high-quality data for visualization is consequently on the machine-scale, easily spanning 100s of TBytes of input to generate a single video frame. In situ visualization, the technique to consume the many-node decomposed data in-memory, as exposed by applications, is the dominant workflow. Although in situ visualization has achieved tremendous progress in the last decade, scaling to system-size together with the application codes that produce its data, there is one important question that we cannot skip: is what we produce insightful and inspiring?
△ Less
Submitted 30 September, 2023;
originally announced October 2023.
-
Particle-in-Cell Simulations of Relativistic Magnetic Reconnection with Advanced Maxwell Solver Algorithms
Authors:
Hannah Klion,
Revathi Jambunathan,
Michael E. Rowan,
Eloise Yang,
Donald Willcox,
Jean-Luc Vay,
Remi Lehe,
Andrew Myers,
Axel Huebl,
Weiqun Zhang
Abstract:
Relativistic magnetic reconnection is a non-ideal plasma process that is a source of non-thermal particle acceleration in many high-energy astrophysical systems. Particle-in-cell (PIC) methods are commonly used for simulating reconnection from first principles. While much progress has been made in understanding the physics of reconnection, especially in 2D, the adoption of advanced algorithms and…
▽ More
Relativistic magnetic reconnection is a non-ideal plasma process that is a source of non-thermal particle acceleration in many high-energy astrophysical systems. Particle-in-cell (PIC) methods are commonly used for simulating reconnection from first principles. While much progress has been made in understanding the physics of reconnection, especially in 2D, the adoption of advanced algorithms and numerical techniques for efficiently modeling such systems has been limited. With the GPU-accelerated PIC code WarpX, we explore the accuracy and potential performance benefits of two advanced Maxwell solver algorithms: a non-standard finite difference scheme (CKC) and an ultrahigh-order pseudo-spectral method (PSATD). We find that for the relativistic reconnection problem, CKC and PSATD qualitatively and quantitatively match the standard Yee-grid finite-difference method. CKC and PSATD both admit a time step that is 40% longer than Yee, resulting in a ~40% faster time to solution for CKC, but no performance benefit for PSATD when using a current deposition scheme that satisfies Gauss's law. Relaxing this constraint maintains accuracy and yields a 30% speedup. Unlike Yee and CKC, PSATD is numerically stable at any time step, allowing for a larger time step than with the finite-difference methods. We found that increasing the time step 2.4-3 times over the standard Yee step still yields accurate results, but only translates to modest performance improvements over CKC due to the current deposition scheme used with PSATD. Further optimization of this scheme will likely improve the effective performance of PSATD.
△ Less
Submitted 20 April, 2023;
originally announced April 2023.
-
From Compact Plasma Particle Sources to Advanced Accelerators with Modeling at Exascale
Authors:
Axel Huebl,
Remi Lehe,
Edoardo Zoni,
Olga Shapoval,
Ryan T. Sandberg,
Marco Garten,
Arianna Formenti,
Revathi Jambunathan,
Prabhat Kumar,
Kevin Gott,
Andrew Myers,
Weiqun Zhang,
Ann Almgren,
Chad E. Mitchell,
Ji Qiang,
David Grote,
Alexander Sinn,
Severin Diederichs,
Maxence Thevenet,
Luca Fedeli,
Thomas Clark,
Neil Zaim,
Henri Vincenti,
Jean-Luc Vay
Abstract:
Developing complex, reliable advanced accelerators requires a coordinated, extensible, and comprehensive approach in modeling, from source to the end of beam lifetime. We present highlights in Exascale Computing to scale accelerator modeling software to the requirements set for contemporary science drivers. In particular, we present the first laser-plasma modeling on an exaflop supercomputer using…
▽ More
Developing complex, reliable advanced accelerators requires a coordinated, extensible, and comprehensive approach in modeling, from source to the end of beam lifetime. We present highlights in Exascale Computing to scale accelerator modeling software to the requirements set for contemporary science drivers. In particular, we present the first laser-plasma modeling on an exaflop supercomputer using the US DOE Exascale Computing Project WarpX. Leveraging developments for Exascale, the new DOE SCIDAC-5 Consortium for Advanced Modeling of Particle Accelerators (CAMPA) will advance numerical algorithms and accelerate community modeling codes in a cohesive manner: from beam source, over energy boost, transport, injection, storage, to application or interaction. Such start-to-end modeling will enable the exploration of hybrid accelerators, with conventional and advanced elements, as the next step for advanced accelerator modeling. Following open community standards, we seed an open ecosystem of codes that can be readily combined with each other and machine learning frameworks. These will cover ultrafast to ultraprecise modeling for future hybrid accelerator design, even enabling virtual test stands and twins of accelerators that can be used in operations.
△ Less
Submitted 18 April, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
-
Next Generation Computational Tools for the Modeling and Design of Particle Accelerators at Exascale
Authors:
Axel Huebl,
Remi Lehe,
Chad E. Mitchell,
Ji Qiang,
Robert D. Ryne,
Ryan T. Sandberg,
Jean-Luc Vay
Abstract:
Particle accelerators are among the largest, most complex devices. To meet the challenges of increasing energy, intensity, accuracy, compactness, complexity and efficiency, increasingly sophisticated computational tools are required for their design and optimization. It is key that contemporary software take advantage of the latest advances in computer hardware and scientific software engineering…
▽ More
Particle accelerators are among the largest, most complex devices. To meet the challenges of increasing energy, intensity, accuracy, compactness, complexity and efficiency, increasingly sophisticated computational tools are required for their design and optimization. It is key that contemporary software take advantage of the latest advances in computer hardware and scientific software engineering practices, delivering speed, reproducibility and feature composability for the aforementioned challenges. A new open source software stack is being developed at the heart of the Beam pLasma Accelerator Simulation Toolkit (BLAST) by LBNL and collaborators, providing new particle-in-cell modeling codes capable of exploiting the power of GPUs on Exascale supercomputers. Combined with advanced numerical techniques, such as mesh-refinement, and intrinsic support for machine learning, these codes are primed to provide ultrafast to ultraprecise modeling for future accelerator design and operations.
△ Less
Submitted 9 August, 2022; v1 submitted 3 August, 2022;
originally announced August 2022.
-
Improving I/O Performance for Exascale Applications through Online Data Layout Reorganization
Authors:
Lipeng Wan,
Axel Huebl,
Junmin Gu,
Franz Poeschel,
Ana Gainaru,
Ruonan Wang,
Jieyang Chen,
Xin Liang,
Dmitry Ganyushin,
Todd Munson,
Ian Foster,
Jean-Luc Vay,
Norbert Podhorszki,
Kesheng Wu,
Scott Klasky
Abstract:
The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Y…
▽ More
The applications being developed within the U.S. Exascale Computing Project (ECP) to run on imminent Exascale computers will generate scientific results with unprecedented fidelity and record turn-around time. Many of these codes are based on particle-mesh methods and use advanced algorithms, especially dynamic load-balancing and mesh-refinement, to achieve high performance on Exascale machines. Yet, as such algorithms improve parallel application efficiency, they raise new challenges for I/O logic due to their irregular and dynamic data distributions. Thus, while the enormous data rates of Exascale simulations already challenge existing file system write strategies, the need for efficient read and processing of generated data introduces additional constraints on the data layout strategies that can be used when writing data to secondary storage. We review these I/O challenges and introduce two online data layout reorganization approaches for achieving good tradeoffs between read and write performance. We demonstrate the benefits of using these two approaches for the ECP particle-in-cell simulation WarpX, which serves as a motif for a large class of important Exascale applications. We show that by understanding application I/O patterns and carefully designing data layouts we can increase read performance by more than 80%.
△ Less
Submitted 15 July, 2021;
originally announced July 2021.
-
In-Situ Assessment of Device-Side Compute Work for Dynamic Load Balancing in a GPU-Accelerated PIC Code
Authors:
Michael E. Rowan,
Axel Huebl,
Kevin N. Gott,
Jack Deslippe,
Maxence Thévenet,
Remi Lehe,
Jean-Luc Vay
Abstract:
Maintaining computational load balance is important to the performant behavior of codes which operate under a distributed computing model. This is especially true for GPU architectures, which can suffer from memory oversubscription if improperly load balanced. We present enhancements to traditional load balancing approaches and explicitly target GPU architectures, exploring the resulting performan…
▽ More
Maintaining computational load balance is important to the performant behavior of codes which operate under a distributed computing model. This is especially true for GPU architectures, which can suffer from memory oversubscription if improperly load balanced. We present enhancements to traditional load balancing approaches and explicitly target GPU architectures, exploring the resulting performance. A key component of our enhancements is the introduction of several GPU-amenable strategies for assessing compute work. These strategies are implemented and benchmarked to find the most optimal data collection methodology for in-situ assessment of GPU compute work. For the fully kinetic particle-in-cell code WarpX, which supports MPI+CUDA parallelism, we investigate the performance of the improved dynamic load balancing via a strong scaling-based performance model and show that, for a laser-ion acceleration test problem run with up to 6144 GPUs on Summit, the enhanced dynamic load balancing achieves from 62%--74% (88% when running on 6 GPUs) of the theoretically predicted maximum speedup; for the 96-GPU case, we find that dynamic load balancing improves performance relative to baselines without load balancing (3.8x speedup) and with static load balancing (1.2x speedup). Our results provide important insights into dynamic load balancing and performance assessment, and are particularly relevant in the context of distributed memory applications ran on GPUs.
△ Less
Submitted 22 April, 2021;
originally announced April 2021.
-
Porting WarpX to GPU-accelerated platforms
Authors:
A. Myers,
A. Almgren,
L. D. Amorim,
J. Bell,
L. Fedeli,
L. Ge,
K. Gott,
D. P. Grote,
M. Hogan,
A. Huebl,
R. Jambunathan,
R. Lehe,
C. Ng,
M. Rowan,
O. Shapoval,
M. Thévenet,
J. -L. Vay,
H. Vincenti,
E. Yang,
N. Zaïm,
W. Zhang,
Y. Zhao,
E. Zoni
Abstract:
WarpX is a general purpose electromagnetic particle-in-cell code that was originally designed to run on many-core CPU architectures. We describe the strategy followed to allow WarpX to use the GPU-accelerated nodes on OLCF's Summit supercomputer, a strategy we believe will extend to the upcoming machines Frontier and Aurora. We summarize the challenges encountered, lessons learned, and give curren…
▽ More
WarpX is a general purpose electromagnetic particle-in-cell code that was originally designed to run on many-core CPU architectures. We describe the strategy followed to allow WarpX to use the GPU-accelerated nodes on OLCF's Summit supercomputer, a strategy we believe will extend to the upcoming machines Frontier and Aurora. We summarize the challenges encountered, lessons learned, and give current performance results on a series of relevant benchmark problems.
△ Less
Submitted 2 September, 2021; v1 submitted 28 January, 2021;
originally announced January 2021.
-
High Energy Physics Forum for Computational Excellence: Working Group Reports (I. Applications Software II. Software Libraries and Tools III. Systems)
Authors:
Salman Habib,
Robert Roser,
Tom LeCompte,
Zach Marshall,
Anders Borgland,
Brett Viren,
Peter Nugent,
Makoto Asai,
Lothar Bauerdick,
Hal Finkel,
Steve Gottlieb,
Stefan Hoeche,
Paul Sheldon,
Jean-Luc Vay,
Peter Elmer,
Michael Kirby,
Simon Patton,
Maxim Potekhin,
Brian Yanny,
Paolo Calafiura,
Eli Dart,
Oliver Gutsche,
Taku Izubuchi,
Adam Lyon,
Don Petravick
Abstract:
Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence…
▽ More
Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence (HEP-FCE) initiated a roadmap planning activity with two key overlapping drivers -- 1) software effectiveness, and 2) infrastructure and expertise advancement. The HEP-FCE formed three working groups, 1) Applications Software, 2) Software Libraries and Tools, and 3) Systems (including systems software), to provide an overview of the current status of HEP computing and to present findings and opportunities for the desired HEP computational roadmap. The final versions of the reports are combined in this document, and are presented along with introductory material.
△ Less
Submitted 28 October, 2015;
originally announced October 2015.
-
White Paper on DOE-HEP Accelerator Modeling Science Activities
Authors:
Jean-Luc Vay,
Cameron G. R. Geddes,
Alice Koniges,
Alex Friedman,
David P. Grote,
David L. Bruhwiler,
John P. Verboncoeur
Abstract:
Toward the goal of maximizing the impact of computer modeling on the design of future particle accelerators and the development of new accelerator techniques & technologies, this white paper presents the rationale for: (a) strengthening and expanding programmatic activities in accelerator modeling science within the Department of Energy (DOE) Office of High Energy Physics (HEP) and (b) increasing…
▽ More
Toward the goal of maximizing the impact of computer modeling on the design of future particle accelerators and the development of new accelerator techniques & technologies, this white paper presents the rationale for: (a) strengthening and expanding programmatic activities in accelerator modeling science within the Department of Energy (DOE) Office of High Energy Physics (HEP) and (b) increasing the community-wide coordination and integration of code development.
△ Less
Submitted 1 October, 2013; v1 submitted 13 September, 2013;
originally announced September 2013.
