-
PyFR v2.0.3: Towards Industrial Adoption of Scale-Resolving Simulations
Authors:
Freddie D. Witherden,
Peter E. Vincent,
Will Trojak,
Yoshiaki Abe,
Amir Akbarzadeh,
Semih Akkurt,
Mohammad Alhawwary,
Lidia Caros,
Tarik Dzanic,
Giorgio Giangaspero,
Arvind S. Iyer,
Antony Jameson,
Marius Koch,
Niki Loppi,
Sambit Mishra,
Rishit Modi,
Gonzalo Sáez-Mischlich,
Jin Seok Park,
Brian C. Vermeire,
Lai Wang
Abstract:
PyFR is an open-source cross-platform computational fluid dynamics framework based on the high-order Flux Reconstruction approach, specifically designed for undertaking high-accuracy scale-resolving simulations in the vicinity of complex engineering geometries. Since the initial release of PyFR v0.1.0 in 2013, a range of new capabilities have been added to the framework, with a view to enabling in…
▽ More
PyFR is an open-source cross-platform computational fluid dynamics framework based on the high-order Flux Reconstruction approach, specifically designed for undertaking high-accuracy scale-resolving simulations in the vicinity of complex engineering geometries. Since the initial release of PyFR v0.1.0 in 2013, a range of new capabilities have been added to the framework, with a view to enabling industrial adoption of the capability. This paper provides details of those enhancements as released in PyFR v2.0.3, explains efforts to grow an engaged developer and user community, and provides latest performance and scaling results on up to 1024 AMD Instinct MI250X accelerators of Frontier at ORNL (each with two GCDs), and up to 2048 NVIDIA GH200 GPUs on Alps at CSCS.
△ Less
Submitted 29 August, 2024;
originally announced August 2024.
-
Validation of wall boundary conditions for simulating complex fluid flows via the Boltzmann equation: Momentum transport and skin friction
Authors:
Tarik Dzanic,
Freddie D. Witherden,
Luigi Martinelli
Abstract:
The influence and validity of wall boundary conditions for non-equilibrium fluid flows described by the Boltzmann equation remains an open problem. The substantial computational cost of directly solving the Boltzmann equation has limited the extent of numerical validation studies to simple, often two-dimensional, flow problems. Recent algorithmic advancements for the Boltzmann--BGK equation introd…
▽ More
The influence and validity of wall boundary conditions for non-equilibrium fluid flows described by the Boltzmann equation remains an open problem. The substantial computational cost of directly solving the Boltzmann equation has limited the extent of numerical validation studies to simple, often two-dimensional, flow problems. Recent algorithmic advancements for the Boltzmann--BGK equation introduced by the authors, consisting of a high-order spatial discretization augmented with a discretely-conservative velocity model, have made it feasible to accurately simulate unsteady three-dimensional flow problems across both the rarefied and continuum regimes. This work presents a comprehensive evaluation and validation of wall boundary conditions across a variety of flow regimes, primarily for the purpose of exploring their effects on momentum transfer in the low Mach limit. Results are presented for a range of steady and unsteady wall-bounded flow problems across both the rarefied and continuum regimes, from canonical two-dimensional laminar flows to unsteady three-dimensional transitional and turbulent flows, the latter of which are the first instances of wall-bounded turbulent flows computed by directly solving the Boltzmann equation. We show that approximations of the molecular gas dynamics equations can accurately predict both non-equilibrium phenomena and complex hydrodynamic flow instabilities and show how spatial and velocity domain resolution affect the accuracy. The results indicate that an accurate approximation of particle transport (i.e. high spatial resolution) is significantly more important than particle collision (i.e. high velocity domain resolution) for predicting flow instabilities and momentum transfer consistent with that predicted by the hydrodynamic equations and that these effects can be computed accurately even with very few degrees of freedom in the velocity domain.
△ Less
Submitted 31 December, 2023; v1 submitted 1 August, 2023;
originally announced August 2023.
-
On the anti-aliasing properties of entropy filtering for discontinuous spectral element approximations of under-resolved turbulent flows
Authors:
Tarik Dzanic,
Will Trojak,
Freddie D. Witherden
Abstract:
For large Reynolds number flows, it is typically necessary to perform simulations that are under-resolved with respect to the underlying flow physics. For nodal discontinuous spectral element approximations of these under-resolved flows, the collocation projection of the nonlinear flux can introduce aliasing errors which can result in numerical instabilities. In Dzanic and Witherden (J. Comput. Ph…
▽ More
For large Reynolds number flows, it is typically necessary to perform simulations that are under-resolved with respect to the underlying flow physics. For nodal discontinuous spectral element approximations of these under-resolved flows, the collocation projection of the nonlinear flux can introduce aliasing errors which can result in numerical instabilities. In Dzanic and Witherden (J. Comput. Phys., 468, 2022), an entropy-based adaptive filtering approach was introduced as a robust, parameter-free shock-capturing method for discontinuous spectral element methods. This work explores the ability of entropy filtering for mitigating aliasing-driven instabilities in the simulation of under-resolved turbulent flows through high-order implicit large eddy simulations of a NACA0021 airfoil in deep stall at a Reynolds number of 270,000. It was observed that entropy filtering can adequately mitigate aliasing-driven instabilities without degrading the accuracy of the underlying high-order scheme on par with standard anti-aliasing methods such as over-integration, albeit with marginally worse performance at higher approximation orders.
△ Less
Submitted 31 December, 2023; v1 submitted 26 February, 2023;
originally announced February 2023.
-
Positivity-preserving entropy filtering for the ideal magnetohydrodynamics equations
Authors:
Tarik Dzanic,
Freddie D. Witherden
Abstract:
In this work, we present a positivity-preserving adaptive filtering approach for discontinuous spectral element approximations of the ideal magnetohydrodynamics equations. This approach combines the entropy filtering method (Dzanic and Witherden, J. Comput. Phys., 468, 2022) for shock capturing in gas dynamics along with the eight-wave method for enforcing a divergence-free magnetic field. Due to…
▽ More
In this work, we present a positivity-preserving adaptive filtering approach for discontinuous spectral element approximations of the ideal magnetohydrodynamics equations. This approach combines the entropy filtering method (Dzanic and Witherden, J. Comput. Phys., 468, 2022) for shock capturing in gas dynamics along with the eight-wave method for enforcing a divergence-free magnetic field. Due to the inclusion of non-conservative source terms, an operator-splitting approach is introduced to ensure that the positivity and entropy constraints remain satisfied by the discrete solution. Furthermore, a computationally efficient algorithm for solving the optimization process for this nonlinear filtering approach is presented. The resulting scheme can robustly resolve strong discontinuities on general unstructured grids without tunable parameters while recovering high-order accuracy for smooth solutions. The efficacy of the scheme is shown in numerical experiments on various problems including extremely magnetized blast waves and three-dimensional magnetohydrodynamic instabilities.
△ Less
Submitted 17 September, 2023; v1 submitted 8 January, 2023;
originally announced January 2023.
-
A positivity-preserving and conservative high-order flux reconstruction method for the polyatomic Boltzmann--BGK equation
Authors:
Tarik Dzanic,
Freddie D. Witherden,
Luigi Martinelli
Abstract:
In this work, we present a positivity-preserving high-order flux reconstruction method for the polyatomic Boltzmann--BGK equation augmented with a discrete velocity model that ensures the scheme is discretely conservative. Through modeling the internal degrees of freedom, the approach is further extended to polyatomic molecules and can encompass arbitrary constitutive laws. The approach is validat…
▽ More
In this work, we present a positivity-preserving high-order flux reconstruction method for the polyatomic Boltzmann--BGK equation augmented with a discrete velocity model that ensures the scheme is discretely conservative. Through modeling the internal degrees of freedom, the approach is further extended to polyatomic molecules and can encompass arbitrary constitutive laws. The approach is validated on a series of large-scale complex numerical experiments, ranging from shock-dominated flows computed on unstructured grids to direct numerical simulation of three-dimensional compressible turbulent flows, the latter of which is the first instance of such a flow computed by directly solving the Boltzmann equation. The results show the ability of the scheme to directly resolve shock structures without any ad hoc numerical shock capturing method and correctly approximate turbulent flow phenomena in a consistent manner with the hydrodynamic equations.
△ Less
Submitted 18 April, 2023; v1 submitted 27 October, 2022;
originally announced October 2022.
-
Positivity-Preserving Entropy-Based Adaptive Filtering for Discontinuous Spectral Element Methods
Authors:
Tarik Dzanic,
Freddie D. Witherden
Abstract:
In this work, we present a positivity-preserving entropy-based adaptive filtering method for shock capturing in discontinuous spectral element methods. By adapting the filter strength to enforce positivity and a local discrete minimum entropy principle, the resulting approach can robustly resolve strong discontinuities with sub-element resolution, does not require problem-dependent parameter tunin…
▽ More
In this work, we present a positivity-preserving entropy-based adaptive filtering method for shock capturing in discontinuous spectral element methods. By adapting the filter strength to enforce positivity and a local discrete minimum entropy principle, the resulting approach can robustly resolve strong discontinuities with sub-element resolution, does not require problem-dependent parameter tuning, and can be easily implemented on general unstructured meshes with relatively low computational cost. The efficacy of the approach is shown in numerical experiments on hyperbolic and mixed hyperbolic-parabolic conservation laws such as the Euler and Navier-Stokes equations for problems including extreme shocks, shock-vortex interactions, and complex compressible turbulent flows.
△ Less
Submitted 28 July, 2022; v1 submitted 25 January, 2022;
originally announced January 2022.
-
Utilizing Time-Reversibility for Shock Capturing in Nonlinear Hyperbolic Conservation Laws
Authors:
Tarik Dzanic,
Will Trojak,
Freddie D. Witherden
Abstract:
In this work, we introduce a novel approach to formulating an artificial viscosity for shock capturing in nonlinear hyperbolic systems by utilizing the property that the solutions of hyperbolic conservation laws are not reversible in time in the vicinity of shocks. The proposed approach does not require any additional governing equations or a priori knowledge of the hyperbolic system in question,…
▽ More
In this work, we introduce a novel approach to formulating an artificial viscosity for shock capturing in nonlinear hyperbolic systems by utilizing the property that the solutions of hyperbolic conservation laws are not reversible in time in the vicinity of shocks. The proposed approach does not require any additional governing equations or a priori knowledge of the hyperbolic system in question, is independent of the mesh and approximation order, and requires the use of only one tunable parameter. The primary novelty is that the resulting artificial viscosity is unique for each component of the conservation law which is advantageous for systems in which some components exhibit discontinuities while others do not. The efficacy of the method is shown in numerical experiments of multi-dimensional hyperbolic conservation laws such as nonlinear transport, Euler equations, and ideal magnetohydrodynamics using a high-order discontinuous spectral element method on unstructured grids.
△ Less
Submitted 18 April, 2022; v1 submitted 7 October, 2021;
originally announced October 2021.
-
On Fourier analysis of polynomial multigrid for arbitrary multi-stage cycles
Authors:
Will Trojak,
Freddie D. Witherden
Abstract:
The Fourier analysis of the \emph{p}-multigrid acceleration technique is considered for a dual-time scheme applied to the advection-diffusion equation with various cycle configurations. It is found that improved convergence can be achieved through \emph{V}-cycle asymmetry where additional prolongation smoothing is applied. Experiments conducted on the artificial compressibility formulation of the…
▽ More
The Fourier analysis of the \emph{p}-multigrid acceleration technique is considered for a dual-time scheme applied to the advection-diffusion equation with various cycle configurations. It is found that improved convergence can be achieved through \emph{V}-cycle asymmetry where additional prolongation smoothing is applied. Experiments conducted on the artificial compressibility formulation of the Navier--Stokes equations found that these analytic findings could be observed numerically in the pressure residual, whereas velocity terms---which are more hyperbolic in character---benefited primarily from increased pseudo-time steps.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Turbulence closure modeling with data-driven techniques: physical compatibility and consistency considerations
Authors:
Salar Taghizadeh,
Freddie D. Witherden,
Sharath S. Girimaji
Abstract:
A recent thrust in turbulence closure modeling research is to incorporate machine learning (ML) elements, such as neural networks, for the purpose of enhancing the predictive capability to a broader class of flows. Such a turbulence closure framework entails solving a system of equations comprised of ML functionals coupled with traditional (physics-based - PB) elements. While combining closure ele…
▽ More
A recent thrust in turbulence closure modeling research is to incorporate machine learning (ML) elements, such as neural networks, for the purpose of enhancing the predictive capability to a broader class of flows. Such a turbulence closure framework entails solving a system of equations comprised of ML functionals coupled with traditional (physics-based - PB) elements. While combining closure elements from fundamentally different ideologies can lead to unprecedented progress, there are many critical challenges that must be overcome. This study examines three such challenges: (i) Physical compatibility (or lack thereof) between ML and PB constituents of the modeling system of equations; (ii) Internal (self) consistency of the ML training process; and (iii) Formulation of an optimal objective (or loss) function for training. These issues are critically important for generalization of the ML-enhanced methods to predictive computations of complex engineering flows. Training and implementation strategies in current practice that may lead to significant incompatibilities and inconsistencies are identified. Using the simple test case of turbulent channel flow, key deficiencies are highlighted and proposals for mitigating them are investigated. Compatibility constraints are evaluated and it is demonstrated that an iterative training procedure can help ensure certain degree of consistency. In summary, this work develops foundational tenets to guide development of ML-enhanced turbulence closure models.
△ Less
Submitted 6 April, 2020;
originally announced April 2020.
-
Parameter-Conditioned Sequential Generative Modeling of Fluid Flows
Authors:
Jeremy Morton,
Freddie D. Witherden,
Mykel J. Kochenderfer
Abstract:
The computational cost associated with simulating fluid flows can make it infeasible to run many simulations across multiple flow conditions. Building upon concepts from generative modeling, we introduce a new method for learning neural network models capable of performing efficient parameterized simulations of fluid flows. Evaluated on their ability to simulate both two-dimensional and three-dime…
▽ More
The computational cost associated with simulating fluid flows can make it infeasible to run many simulations across multiple flow conditions. Building upon concepts from generative modeling, we introduce a new method for learning neural network models capable of performing efficient parameterized simulations of fluid flows. Evaluated on their ability to simulate both two-dimensional and three-dimensional fluid flows, trained models are shown to capture local and global properties of the flow fields at a wide array of flow conditions. Furthermore, flow simulations generated by the trained models are shown to be orders of magnitude faster than the corresponding computational fluid dynamics simulations.
△ Less
Submitted 13 December, 2019;
originally announced December 2019.
-
Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning
Authors:
Kevin T. Carlberg,
Antony Jameson,
Mykel J. Kochenderfer,
Jeremy Morton,
Liqian Peng,
Freddie D. Witherden
Abstract:
Data I/O poses a significant bottleneck in large-scale CFD simulations; thus, practitioners would like to significantly reduce the number of times the solution is saved to disk, yet retain the ability to recover any field quantity (at any time instance) a posteriori. The objective of this work is therefore to accurately recover missing CFD data a posteriori at any time instance, given that the sol…
▽ More
Data I/O poses a significant bottleneck in large-scale CFD simulations; thus, practitioners would like to significantly reduce the number of times the solution is saved to disk, yet retain the ability to recover any field quantity (at any time instance) a posteriori. The objective of this work is therefore to accurately recover missing CFD data a posteriori at any time instance, given that the solution has been written to disk at only a relatively small number of time instances. We consider in particular high-order discretizations (e.g., discontinuous Galerkin), as such techniques are becoming increasingly popular for the simulation of highly separated flows. To satisfy this objective, this work proposes a methodology consisting of two stages: 1) dimensionality reduction and 2) dynamics learning. For dimensionality reduction, we propose a novel hierarchical approach. First, the method reduces the number of degrees of freedom within each element of the high-order discretization by applying autoencoders from deep learning. Second, the methodology applies principal component analysis to compress the global vector of encodings. This leads to a low-dimensional state, which associates with a nonlinear embedding of the original CFD data. For dynamics learning, we propose to apply regression techniques (e.g., kernel methods) to learn the discrete-time velocity characterizing the time evolution of this low-dimensional state. A numerical example on a large-scale CFD example characterized by nearly 13 million degrees of freedom illustrates the suitability of the proposed method in an industrial setting.
△ Less
Submitted 28 May, 2019; v1 submitted 30 November, 2018;
originally announced December 2018.
-
A Parallel Direct Cut Algorithm for High-Order Overset Methods with Application to a Spinning Golf Ball
Authors:
Jacob A. Crabill,
Freddie D. Witherden,
Antony Jameson
Abstract:
Overset methods are commonly employed to enable the effective simulation of problems involving complex geometries and moving objects such as rotorcraft. This paper presents a novel overset domain connectivity algorithm based upon the direct cut approach suitable for use with GPU-accelerated solvers on high-order curved grids. In contrast to previous methods it is capable of exploiting the highly d…
▽ More
Overset methods are commonly employed to enable the effective simulation of problems involving complex geometries and moving objects such as rotorcraft. This paper presents a novel overset domain connectivity algorithm based upon the direct cut approach suitable for use with GPU-accelerated solvers on high-order curved grids. In contrast to previous methods it is capable of exploiting the highly data-parallel nature of modern accelerators. Further, the approach is also substantially more efficient at handling the curved grids which arise within the context of high-order methods. An implementation of this new algorithm is presented and combined with a high-order fluid dynamics code. The algorithm is validated against several benchmark problems, including flow over a spinning golf ball at a Reynolds number of 150,000.
△ Less
Submitted 23 May, 2018; v1 submitted 21 November, 2017;
originally announced November 2017.
-
Heterogeneous Computing on Mixed Unstructured Grids with PyFR
Authors:
F. D. Witherden,
B. C. Vermeire,
P. E. Vincent
Abstract:
PyFR is an open-source high-order accurate computational fluid dynamics solver for mixed unstructured grids that can target a range of hardware platforms from a single codebase. In this paper we demonstrate the ability of PyFR to perform high-order accurate unsteady simulations of flow on mixed unstructured grids using heterogeneous multi-node hardware. Specifically, after benchmarking single-node…
▽ More
PyFR is an open-source high-order accurate computational fluid dynamics solver for mixed unstructured grids that can target a range of hardware platforms from a single codebase. In this paper we demonstrate the ability of PyFR to perform high-order accurate unsteady simulations of flow on mixed unstructured grids using heterogeneous multi-node hardware. Specifically, after benchmarking single-node performance for various platforms, PyFR v0.2.2 is used to undertake simulations of unsteady flow over a circular cylinder at Reynolds number 3 900 using a mixed unstructured grid of prismatic and tetrahedral elements on a desktop workstation containing an Intel Xeon E5-2697 v2 CPU, an NVIDIA Tesla K40c GPU, and an AMD FirePro W9100 GPU. Both the performance and accuracy of PyFR are assessed. PyFR v0.2.2 is freely available under a 3-Clause New Style BSD license (see www.pyfr.org).
△ Less
Submitted 1 September, 2014;
originally announced September 2014.
-
PyFR: An Open Source Framework for Solving Advection-Diffusion Type Problems on Streaming Architectures using the Flux Reconstruction Approach
Authors:
Freddie D Witherden,
Antony M Farrington,
Peter E Vincent
Abstract:
High-order numerical methods for unstructured grids combine the superior accuracy of high-order spectral or finite difference methods with the geometric flexibility of low-order finite volume or finite element schemes. The Flux Reconstruction (FR) approach unifies various high-order schemes for unstructured grids within a single framework. Additionally, the FR approach exhibits a significant degre…
▽ More
High-order numerical methods for unstructured grids combine the superior accuracy of high-order spectral or finite difference methods with the geometric flexibility of low-order finite volume or finite element schemes. The Flux Reconstruction (FR) approach unifies various high-order schemes for unstructured grids within a single framework. Additionally, the FR approach exhibits a significant degree of element locality, and is thus able to run efficiently on modern streaming architectures, such as Graphical Processing Units (GPUs). The aforementioned properties of FR mean it offers a promising route to performing affordable, and hence industrially relevant, scale-resolving simulations of hitherto intractable unsteady flows within the vicinity of real-world engineering geometries. In this paper we present PyFR, an open-source Python based framework for solving advection-diffusion type problems on streaming architectures using the FR approach. The framework is designed to solve a range of governing systems on mixed unstructured grids containing various element types. It is also designed to target a range of hardware platforms via use of an in-built domain specific language based on the Mako templating engine. The current release of PyFR is able to solve the compressible Euler and Navier-Stokes equations on grids of quadrilateral and triangular elements in two dimensions, and hexahedral elements in three dimensions, targeting clusters of CPUs, and NVIDIA GPUs. Results are presented for various benchmark flow problems, single-node performance is discussed, and scalability of the code is demonstrated on up to 104 NVIDIA M2090 GPUs. The software is freely available under a 3-Clause New Style BSD license (see www.pyfr.org).
△ Less
Submitted 7 May, 2014; v1 submitted 5 December, 2013;
originally announced December 2013.
