-
Representation Meets Optimization: Training PINNs and PIKANs for Gray-Box Discovery in Systems Pharmacology
Authors:
Nazanin Ahmadi Daryakenari,
Khemraj Shukla,
George Em Karniadakis
Abstract:
Physics-Informed Kolmogorov-Arnold Networks (PIKANs) are gaining attention as an effective counterpart to the original multilayer perceptron-based Physics-Informed Neural Networks (PINNs). Both representation models can address inverse problems and facilitate gray-box system identification. However, a comprehensive understanding of their performance in terms of accuracy and speed remains underexpl…
▽ More
Physics-Informed Kolmogorov-Arnold Networks (PIKANs) are gaining attention as an effective counterpart to the original multilayer perceptron-based Physics-Informed Neural Networks (PINNs). Both representation models can address inverse problems and facilitate gray-box system identification. However, a comprehensive understanding of their performance in terms of accuracy and speed remains underexplored. In particular, we introduce a modified PIKAN architecture, tanh-cPIKAN, which is based on Chebyshev polynomials for parametrization of the univariate functions with an extra nonlinearity for enhanced performance. We then present a systematic investigation of how choices of the optimizer, representation, and training configuration influence the performance of PINNs and PIKANs in the context of systems pharmacology modeling. We benchmark a wide range of first-order, second-order, and hybrid optimizers, including various learning rate schedulers. We use the new Optax library to identify the most effective combinations for learning gray-boxes under ill-posed, non-unique, and data-sparse conditions. We examine the influence of model architecture (MLP vs. KAN), numerical precision (single vs. double), the need for warm-up phases for second-order methods, and sensitivity to the initial learning rate. We also assess the optimizer scalability for larger models and analyze the trade-offs introduced by JAX in terms of computational efficiency and numerical accuracy. Using two representative systems pharmacology case studies - a pharmacokinetics model and a chemotherapy drug-response model - we offer practical guidance on selecting optimizers and representation models/architectures for robust and efficient gray-box discovery. Our findings provide actionable insights for improving the training of physics-informed networks in biomedical applications and beyond.
△ Less
Submitted 9 April, 2025;
originally announced April 2025.
-
From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning
Authors:
Juan Diego Toscano,
Vivek Oommen,
Alan John Varghese,
Zongren Zou,
Nazanin Ahmadi Daryakenari,
Chenxi Wu,
George Em Karniadakis
Abstract:
Physics-Informed Neural Networks (PINNs) have emerged as a key tool in Scientific Machine Learning since their introduction in 2017, enabling the efficient solution of ordinary and partial differential equations using sparse measurements. Over the past few years, significant advancements have been made in the training and optimization of PINNs, covering aspects such as network architectures, adapt…
▽ More
Physics-Informed Neural Networks (PINNs) have emerged as a key tool in Scientific Machine Learning since their introduction in 2017, enabling the efficient solution of ordinary and partial differential equations using sparse measurements. Over the past few years, significant advancements have been made in the training and optimization of PINNs, covering aspects such as network architectures, adaptive refinement, domain decomposition, and the use of adaptive weights and activation functions. A notable recent development is the Physics-Informed Kolmogorov-Arnold Networks (PIKANS), which leverage a representation model originally proposed by Kolmogorov in 1957, offering a promising alternative to traditional PINNs. In this review, we provide a comprehensive overview of the latest advancements in PINNs, focusing on improvements in network design, feature expansion, optimization techniques, uncertainty quantification, and theoretical insights. We also survey key applications across a range of fields, including biomedicine, fluid and solid mechanics, geophysics, dynamical systems, heat transfer, chemical engineering, and beyond. Finally, we review computational frameworks and software tools developed by both academia and industry to support PINN research and applications.
△ Less
Submitted 21 October, 2024; v1 submitted 17 October, 2024;
originally announced October 2024.
-
CMINNs: Compartment Model Informed Neural Networks -- Unlocking Drug Dynamics
Authors:
Nazanin Ahmadi Daryakenari,
Shupeng Wang,
George Karniadakis
Abstract:
In the field of pharmacokinetics and pharmacodynamics (PKPD) modeling, which plays a pivotal role in the drug development process, traditional models frequently encounter difficulties in fully encapsulating the complexities of drug absorption, distribution, and their impact on targets. Although multi-compartment models are frequently utilized to elucidate intricate drug dynamics, they can also be…
▽ More
In the field of pharmacokinetics and pharmacodynamics (PKPD) modeling, which plays a pivotal role in the drug development process, traditional models frequently encounter difficulties in fully encapsulating the complexities of drug absorption, distribution, and their impact on targets. Although multi-compartment models are frequently utilized to elucidate intricate drug dynamics, they can also be overly complex. To generalize modeling while maintaining simplicity, we propose an innovative approach that enhances PK and integrated PK-PD modeling by incorporating fractional calculus or time-varying parameter(s), combined with constant or piecewise constant parameters. These approaches effectively model anomalous diffusion, thereby capturing drug trapping and escape rates in heterogeneous tissues, which is a prevalent phenomenon in drug dynamics. Furthermore, this method provides insight into the dynamics of drug in cancer in multi-dose administrations. Our methodology employs a Physics-Informed Neural Network (PINN) and fractional Physics-Informed Neural Networks (fPINNs), integrating ordinary differential equations (ODEs) with integer/fractional derivative order from compartmental modeling with neural networks. This integration optimizes parameter estimation for variables that are time-variant, constant, piecewise constant, or related to the fractional derivative order. The results demonstrate that this methodology offers a robust framework that not only markedly enhances the model's depiction of drug absorption rates and distributed delayed responses but also unlocks different drug-effect dynamics, providing new insights into absorption rates, anomalous diffusion, drug resistance, peristance and pharmacokinetic tolerance, all within a system of just two (fractional) ODEs with explainable results.
△ Less
Submitted 19 September, 2024;
originally announced September 2024.
-
State-space models are accurate and efficient neural operators for dynamical systems
Authors:
Zheyuan Hu,
Nazanin Ahmadi Daryakenari,
Qianli Shen,
Kenji Kawaguchi,
George Em Karniadakis
Abstract:
Physics-informed machine learning (PIML) has emerged as a promising alternative to classical methods for predicting dynamical systems, offering faster and more generalizable solutions. However, existing models, including recurrent neural networks (RNNs), transformers, and neural operators, face challenges such as long-time integration, long-range dependencies, chaotic dynamics, and extrapolation,…
▽ More
Physics-informed machine learning (PIML) has emerged as a promising alternative to classical methods for predicting dynamical systems, offering faster and more generalizable solutions. However, existing models, including recurrent neural networks (RNNs), transformers, and neural operators, face challenges such as long-time integration, long-range dependencies, chaotic dynamics, and extrapolation, to name a few. To this end, this paper introduces state-space models implemented in Mamba for accurate and efficient dynamical system operator learning. Mamba addresses the limitations of existing architectures by dynamically capturing long-range dependencies and enhancing computational efficiency through reparameterization techniques. To extensively test Mamba and compare against another 11 baselines, we introduce several strict extrapolation testbeds that go beyond the standard interpolation benchmarks. We demonstrate Mamba's superior performance in both interpolation and challenging extrapolation tasks. Mamba consistently ranks among the top models while maintaining the lowest computational cost and exceptional extrapolation capabilities. Moreover, we demonstrate the good performance of Mamba for a real-world application in quantitative systems pharmacology for assessing the efficacy of drugs in tumor growth under limited data scenarios. Taken together, our findings highlight Mamba's potential as a powerful tool for advancing scientific machine learning in dynamical systems modeling. (The code will be available at https://github.com/zheyuanhu01/State_Space_Model_Neural_Operator upon acceptance.)
△ Less
Submitted 27 January, 2025; v1 submitted 4 September, 2024;
originally announced September 2024.
-
AI-Aristotle: A Physics-Informed framework for Systems Biology Gray-Box Identification
Authors:
Nazanin Ahmadi Daryakenari,
Mario De Florio,
Khemraj Shukla,
George Em Karniadakis
Abstract:
Discovering mathematical equations that govern physical and biological systems from observed data is a fundamental challenge in scientific research. We present a new physics-informed framework for parameter estimation and missing physics identification (gray-box) in the field of Systems Biology. The proposed framework -- named AI-Aristotle -- combines eXtreme Theory of Functional Connections (X-TF…
▽ More
Discovering mathematical equations that govern physical and biological systems from observed data is a fundamental challenge in scientific research. We present a new physics-informed framework for parameter estimation and missing physics identification (gray-box) in the field of Systems Biology. The proposed framework -- named AI-Aristotle -- combines eXtreme Theory of Functional Connections (X-TFC) domain-decomposition and Physics-Informed Neural Networks (PINNs) with symbolic regression (SR) techniques for parameter discovery and gray-box identification. We test the accuracy, speed, flexibility and robustness of AI-Aristotle based on two benchmark problems in Systems Biology: a pharmacokinetics drug absorption model, and an ultradian endocrine model for glucose-insulin interactions. We compare the two machine learning methods (X-TFC and PINNs), and moreover, we employ two different symbolic regression techniques to cross-verify our results. While the current work focuses on the performance of AI-Aristotle based on synthetic data, it can equally handle noisy experimental data and can even be used for black-box identification in just a few minutes on a laptop. More broadly, our work provides insights into the accuracy, cost, scalability, and robustness of integrating neural networks with symbolic regressors, offering a comprehensive guide for researchers tackling gray-box identification challenges in complex dynamical systems in biomedicine and beyond.
△ Less
Submitted 29 September, 2023;
originally announced October 2023.
