-
MOOSE: Enabling Massively Parallel Multiphysics Simulation
Authors:
Cody J. Permann,
Derek R. Gaston,
David Andrs,
Robert W. Carlsen,
Fande Kong,
Alexander D. Lindsay,
Jason M. Miller,
John W. Peterson,
Andrew E. Slaughter,
Roy H. Stogner,
Richard C. Martineau
Abstract:
Harnessing modern parallel computing resources to achieve complex multi-physics simulations is a daunting task. The Multiphysics Object Oriented Simulation Environment (MOOSE) aims to enable such development by providing simplified interfaces for specification of partial differential equations, boundary conditions, material properties, and all aspects of a simulation without the need to consider t…
▽ More
Harnessing modern parallel computing resources to achieve complex multi-physics simulations is a daunting task. The Multiphysics Object Oriented Simulation Environment (MOOSE) aims to enable such development by providing simplified interfaces for specification of partial differential equations, boundary conditions, material properties, and all aspects of a simulation without the need to consider the parallel, adaptive, nonlinear, finite-element solve that is handled internally. Through the use of interfaces and inheritance, each portion of a simulation becomes reusable and composable in a manner that allows disparate research groups to share code and create an ecosystem of growing capability that lowers the barrier for the creation of multiphysics simulation codes. Included within the framework is a unique capability for building multiscale, multiphysics simulations through simultaneous execution of multiple sub-applications with data transfers between the scales. Other capabilities include automatic differentiation, scaling to a large number of processors, hybrid parallelism, and mesh adaptivity. To date, MOOSE-based applications have been created in areas of science and engineering such as nuclear physics, geothermal science, magneto-hydrodynamics, seismic events, compressible and incompressible fluid flow, microstructure evolution, and advanced manufacturing processes.
△ Less
Submitted 11 November, 2019;
originally announced November 2019.
-
A scalable multilevel domain decomposition preconditioner with a subspace-based coarsening algorithm for the neutron transport calculations
Authors:
Fande Kong,
Yaqi Wang,
Derek R. Gaston,
Alexander D. Lindsay,
Cody J. Permann,
Richard C. Martineau
Abstract:
The multigroup neutron transport equations has been widely used to study the interactions of neutrons with their background materials in nuclear reactors. High-resolution simulations of the multigroup neutron transport equations using modern supercomputers require the development of scalable parallel solving techniques. In this paper, we study a scalable transport method for solving the algebraic…
▽ More
The multigroup neutron transport equations has been widely used to study the interactions of neutrons with their background materials in nuclear reactors. High-resolution simulations of the multigroup neutron transport equations using modern supercomputers require the development of scalable parallel solving techniques. In this paper, we study a scalable transport method for solving the algebraic system arising from the discretization of the multigroup neutron transport equations. The proposed transport method consists of a fully coupled Newton solver for the generalized eigenvalue problems and GMRES together with a novel multilevel domain decomposition preconditioner for the Jacobian system. The multilevel preconditioner has been successfully used for many problems, but the construction of coarse spaces for certain problems, especially for unstructured mesh problems, is expensive and often unscalable. We introduce a new subspace-based coarsening algorithm to address this issue by exploring the structure of the matrix in the discretized version of the neutron transport problems. We numerically demonstrate that the proposed transport method is highly scalable with more than 10,000 processor cores for the 3D C5G7 benchmark problem on unstructured meshes with billions of unknowns. Compared with the traditional multilevel domain decomposition method, the new approach equipped with the subspace-based coarsening algorithm is much faster on the construction of coarse spaces.
△ Less
Submitted 18 June, 2019;
originally announced June 2019.
-
A highly parallel multilevel Newton-Krylov-Schwarz method with subspace-based coarsening and partition-based balancing for the multigroup neutron transport equations on 3D unstructured meshes
Authors:
Fande Kong,
Yaqi Wang,
Derek R. Gaston,
Cody J. Permann,
Andrew E. Slaughter,
Alexander D. Lindsay,
Richard C. Martineau
Abstract:
The multigroup neutron transport equations have been widely used to study the motion of neutrons and their interactions with the background materials. Numerical simulation of the multigroup neutron transport equations is computationally challenging because the equations is defined on a high dimensional phase space (1D in energy, 2D in angle, and 3D in spatial space), and furthermore, for realistic…
▽ More
The multigroup neutron transport equations have been widely used to study the motion of neutrons and their interactions with the background materials. Numerical simulation of the multigroup neutron transport equations is computationally challenging because the equations is defined on a high dimensional phase space (1D in energy, 2D in angle, and 3D in spatial space), and furthermore, for realistic applications, the computational spatial domain is complex and the materials are heterogeneous. The multilevel domain decomposition methods is one of the most popular algorithms for solving the multigroup neutron transport equations, but the construction of coarse spaces is expensive and often not strongly scalable when the number of processor cores is large. In this paper, we study a highly parallel multilevel Newton-Krylov-Schwarz method equipped with several novel components, such as subspace-based coarsening, partition-based balancing and hierarchical mesh partitioning, that enable the overall simulation strongly scalable in terms of the compute time. Compared with the traditional coarsening method, the subspace-based coarsening algorithm significantly reduces the cost of the preconditioner setup that is often unscalable. In addition, the partition-based balancing strategy enhances the parallel efficiency of the overall solver by assigning a nearly-equal amount of work to each processor core. The hierarchical mesh partitioning is able to generate a large number of subdomains and meanwhile minimizes the off-node communication. We numerically show that the proposed algorithm is scalable with more than 10,000 processor cores for a realistic application with a few billions unknowns on 3D unstructured meshes.
△ Less
Submitted 8 March, 2019;
originally announced March 2019.
-
A general-purpose hierarchical mesh partitioning method with node balancing strategies for large-scale numerical simulations
Authors:
Fande Kong,
Roy H. Stogner,
Derek R. Gaston,
John W. Peterson,
Cody J. Permann,
Andrew E. Slaughter,
Richard C. Martineau
Abstract:
Large-scale parallel numerical simulations are essential for a wide range of engineering problems that involve complex, coupled physical processes interacting across a broad range of spatial and temporal scales. The data structures involved in such simulations (meshes, sparse matrices, etc.) are frequently represented as graphs, and these graphs must be optimally partitioned across the available c…
▽ More
Large-scale parallel numerical simulations are essential for a wide range of engineering problems that involve complex, coupled physical processes interacting across a broad range of spatial and temporal scales. The data structures involved in such simulations (meshes, sparse matrices, etc.) are frequently represented as graphs, and these graphs must be optimally partitioned across the available computational resources in order for the underlying calculations to scale efficiently. Partitions which minimize the number of graph edges that are cut (edge-cuts) while simultaneously maintaining a balance in the amount of work (i.e. graph nodes) assigned to each processor core are desirable, and the performance of most existing partitioning software begins to degrade in this metric for partitions with more than than $O(10^3)$ processor cores. In this work, we consider a general-purpose hierarchical partitioner which takes into account the existence of multiple processor cores and shared memory in a compute node while partitioning a graph into an arbitrary number of subgraphs. We demonstrate that our algorithms significantly improve the preconditioning efficiency and overall performance of realistic numerical simulations running on up to 32,768 processor cores with nearly $10^9$ unknowns.
△ Less
Submitted 10 October, 2018; v1 submitted 7 September, 2018;
originally announced September 2018.
