-
BHaHAHA: A Fast, Robust Apparent Horizon Finder Library for Numerical Relativity
Authors:
Zachariah B. Etienne,
Thiago Assumpção,
Leonardo Rosa Werneck,
Samuel D. Tootle
Abstract:
Apparent horizon (AH) finders are essential for characterizing black holes and excising their interiors in numerical relativity (NR) simulations. However, open-source AH finders to date are tightly coupled to individual NR codes. We introduce BHaHAHA, the BlackHoles@Home Apparent Horizon Algorithm, the first open-source, infrastructure-agnostic library for AH finding in NR. BHaHAHA implements the…
▽ More
Apparent horizon (AH) finders are essential for characterizing black holes and excising their interiors in numerical relativity (NR) simulations. However, open-source AH finders to date are tightly coupled to individual NR codes. We introduce BHaHAHA, the BlackHoles@Home Apparent Horizon Algorithm, the first open-source, infrastructure-agnostic library for AH finding in NR. BHaHAHA implements the first-ever hyperbolic flow-based approach, recasting the elliptic partial differential equation for a marginally outer trapped surface as a damped nonlinear wave equation. To enhance performance, BHaHAHA incorporates a multigrid-inspired refinement strategy, an over-relaxation technique, and OpenMP parallelization. When compared to a naïve hyperbolic relaxation implementation, these enhancements result in 64x speedups for difficult common-horizon finds on a single spacetime slice, enabling BHaHAHA to achieve runtimes within 10% of the widely used (single-core) AHFinderDirect and outperform it on multiple cores. For dynamic horizon tracking with typical core counts on a high-performance-computing cluster, BHaHAHA is approximately 2.1 times faster than AHFinderDirect at accuracies limited by interpolation of metric data from the host NR code. Implemented and tested in both the Einstein Toolkit and BlackHoles@Home, BHaHAHA demonstrates that hyperbolic relaxation can be a robust, versatile, and performant approach for AH finding.
△ Less
Submitted 21 May, 2025;
originally announced May 2025.
-
Accelerating Numerical Relativity with Code Generation: CUDA-enabled Hyperbolic Relaxation
Authors:
Samuel D. Tootle,
Leonardo R. Werneck,
Thiago Assumpção,
Terrence Pierre Jacques,
Zachariah B. Etienne
Abstract:
Next-generation gravitational wave detectors such as Cosmic Explorer, the Einstein Telescope, and LISA, demand highly accurate and extensive gravitational wave (GW) catalogs to faithfully extract physical parameters from observed signals. However, numerical relativity (NR) faces significant challenges in generating these catalogs at the required scale and accuracy on modern computers, as NR codes…
▽ More
Next-generation gravitational wave detectors such as Cosmic Explorer, the Einstein Telescope, and LISA, demand highly accurate and extensive gravitational wave (GW) catalogs to faithfully extract physical parameters from observed signals. However, numerical relativity (NR) faces significant challenges in generating these catalogs at the required scale and accuracy on modern computers, as NR codes do not fully exploit modern GPU capabilities. In response, we extend NRPy, a Python-based NR code-generation framework, to develop NRPyEllipticGPU -- a CUDA-optimized elliptic solver tailored for the binary black hole (BBH) initial data problem. NRPyEllipticGPU is the first GPU-enabled elliptic solver in the NR community, supporting a variety of coordinate systems and demonstrating substantial performance improvements on both consumer-grade and HPC-grade GPUs. We show that, when compared to a high-end CPU, NRPyEllipticGPU achieves on a high-end GPU up to a sixteenfold speedup in single precision while increasing double-precision performance by a factor of 2--4. This performance boost leverages the GPU's superior parallelism and memory bandwidth to achieve a compute-bound application and enhancing the overall simulation efficiency. As NRPyEllipticGPU shares the core infrastructure common to NR codes, this work serves as a practical guide for developing full, CUDA-optimized NR codes.
△ Less
Submitted 19 May, 2025; v1 submitted 23 January, 2025;
originally announced January 2025.
-
Addition of tabulated equation of state and neutrino leakage support to IllinoisGRMHD
Authors:
Leonardo R. Werneck,
Zachariah B. Etienne,
Ariadna Murguia-Berthier,
Roland Haas,
Federico Cipolletta,
Scott C. Noble,
Lorenzo Ennoggi,
Federico G. Lopez Armengol,
Bruno Giacomazzo,
Thiago Assumpção,
Joshua Faber,
Tanmayee Gupte,
Bernard J. Kelly,
Julian H. Krolik
Abstract:
We have added support for realistic, microphysical, finite-temperature equations of state (EOS) and neutrino physics via a leakage scheme to IllinoisGRMHD, an open-source GRMHD code for dynamical spacetimes in the Einstein Toolkit. These new features are provided by two new, NRPy+-based codes: NRPyEOS, which performs highly efficient EOS table lookups and interpolations, and NRPyLeakage, which imp…
▽ More
We have added support for realistic, microphysical, finite-temperature equations of state (EOS) and neutrino physics via a leakage scheme to IllinoisGRMHD, an open-source GRMHD code for dynamical spacetimes in the Einstein Toolkit. These new features are provided by two new, NRPy+-based codes: NRPyEOS, which performs highly efficient EOS table lookups and interpolations, and NRPyLeakage, which implements a new, AMR-capable neutrino leakage scheme in the Einstein Toolkit. We have performed a series of strenuous validation tests that demonstrate the robustness of these new codes, particularly on the Cartesian AMR grids provided by Carpet. Furthermore, we show results from fully dynamical GRMHD simulations of single unmagnetized neutron stars, and magnetized binary neutron star mergers. This new version of IllinoisGRMHD, as well as NRPyEOS and NRPyLeakage, is pedagogically documented in Jupyter notebooks and fully open source. The codes will be proposed for inclusion in an upcoming version of the Einstein Toolkit.
△ Less
Submitted 14 December, 2023; v1 submitted 30 August, 2022;
originally announced August 2022.
-
NRPyElliptic: A Fast Hyperbolic Relaxation Elliptic Solver for Numerical Relativity, I: Conformally Flat, Binary Puncture Initial Data
Authors:
Thiago Assumpcao,
Leonardo R. Werneck,
Terrence Pierre Jacques,
Zachariah B. Etienne
Abstract:
We introduce NRPyElliptic, an elliptic solver for numerical relativity (NR) built within the NRPy+ framework. As its first application, NRPyElliptic sets up conformally flat, binary black hole (BBH) puncture initial data (ID) on a single numerical domain, similar to the widely used TwoPunctures code. Unlike TwoPunctures, NRPyElliptic employs a hyperbolic relaxation scheme, whereby arbitrary ellipt…
▽ More
We introduce NRPyElliptic, an elliptic solver for numerical relativity (NR) built within the NRPy+ framework. As its first application, NRPyElliptic sets up conformally flat, binary black hole (BBH) puncture initial data (ID) on a single numerical domain, similar to the widely used TwoPunctures code. Unlike TwoPunctures, NRPyElliptic employs a hyperbolic relaxation scheme, whereby arbitrary elliptic PDEs are trivially transformed into a hyperbolic system of PDEs. As consumers of NR ID generally already possess expertise in solving hyperbolic PDEs, they will generally find NRPyElliptic easier to tweak and extend than other NR elliptic solvers. When evolved forward in (pseudo)time, the hyperbolic system exponentially reaches a steady state that solves the elliptic PDEs. Notably NRPyElliptic accelerates the relaxation waves, which makes it many orders of magnitude faster than the usual constant-wavespeed approach. While it is still ${\sim}12$x slower than TwoPunctures at setting up full-3D BBH ID, NRPyElliptic requires only ${\approx}0.3\%$ of the runtime for a full BBH simulation in the Einstein Toolkit. Future work will focus on improving performance and generating other types of ID, such as binary neutron star.
△ Less
Submitted 22 April, 2022; v1 submitted 3 November, 2021;
originally announced November 2021.
-
Black hole binaries: ergoregions, photon surfaces, wave scattering, and quasinormal modes
Authors:
Thiago Assumpcao,
Vitor Cardoso,
Akihiro Ishibashi,
Mauricio Richartz,
Miguel Zilhao
Abstract:
Closed photon orbits around isolated black holes are related to important aspects of black hole physics, such as strong lensing, absorption cross section of null particles and the way that black holes relax through quasinormal ringing. When two black holes are present -- such as during the inspiral and merger events of interest for gravitational-wave detectors -- the concept of closed photon orbit…
▽ More
Closed photon orbits around isolated black holes are related to important aspects of black hole physics, such as strong lensing, absorption cross section of null particles and the way that black holes relax through quasinormal ringing. When two black holes are present -- such as during the inspiral and merger events of interest for gravitational-wave detectors -- the concept of closed photon orbits still exists, but its properties are basically unknown. With these applications in mind, we study here the closed photon orbits of two different static black hole binaries. The first one is the Majumdar-Papapetrou geometry describing two extremal, charged black holes in equilibrium, while the second one is the double sink solution of fluid dynamics, which describes (in a curved-spacetime language) two "dumb" holes. For the latter solution, we also characterize its dynamical response to external perturbations, and study how it relates to the photon orbits. In addition, we compute the ergoregion of such spacetime and show that it does not coincide with the event horizon.
△ Less
Submitted 13 September, 2018; v1 submitted 20 June, 2018;
originally announced June 2018.
