-
Digital Twin Calibration with Model-Based Reinforcement Learning
Authors:
Hua Zheng,
Wei Xie,
Ilya O. Ryzhov,
Keilung Choy
Abstract:
This paper presents a novel methodological framework, called the Actor-Simulator, that incorporates the calibration of digital twins into model-based reinforcement learning for more effective control of stochastic systems with complex nonlinear dynamics. Traditional model-based control often relies on restrictive structural assumptions (such as linear state transitions) and fails to account for pa…
▽ More
This paper presents a novel methodological framework, called the Actor-Simulator, that incorporates the calibration of digital twins into model-based reinforcement learning for more effective control of stochastic systems with complex nonlinear dynamics. Traditional model-based control often relies on restrictive structural assumptions (such as linear state transitions) and fails to account for parameter uncertainty in the model. These issues become particularly critical in industries such as biopharmaceutical manufacturing, where process dynamics are complex and not fully known, and only a limited amount of data is available. Our approach jointly calibrates the digital twin and searches for an optimal control policy, thus accounting for and reducing model error. We balance exploration and exploitation by using policy performance as a guide for data collection. This dual-component approach provably converges to the optimal policy, and outperforms existing methods in extensive numerical experiments based on the biopharmaceutical manufacturing domain.
△ Less
Submitted 4 January, 2025;
originally announced January 2025.
-
Evolution of the star formation rate surface density main sequence. Insights from a semi-analytic simulation since $z = 12$
Authors:
Jakub Nadolny,
Michał J. Michałowski,
Massimiliano Parente,
Martín Solar,
Przemysław Nowaczyk,
Oleh Ryzhov,
Aleksandra Leśniewska
Abstract:
Recent high-redshift ($z>4$) spatially resolved observations with the James Webb Space Telesescope have shown the evolution of the star formation rate (SFR) surface density ($Σ_{\rm SFR}$) and its main sequence in the $Σ_{\rm SFR}$-$M_*$ diagram ($Σ_{\rm SFR}{\rm MS}$). The $Σ_{\rm SFR}{\rm MS}$\ is already observed at cosmic morning ($z\sim7.5$). The use of $Σ_{\rm SFR}$\ is physically motivated…
▽ More
Recent high-redshift ($z>4$) spatially resolved observations with the James Webb Space Telesescope have shown the evolution of the star formation rate (SFR) surface density ($Σ_{\rm SFR}$) and its main sequence in the $Σ_{\rm SFR}$-$M_*$ diagram ($Σ_{\rm SFR}{\rm MS}$). The $Σ_{\rm SFR}{\rm MS}$\ is already observed at cosmic morning ($z\sim7.5$). The use of $Σ_{\rm SFR}$\ is physically motivated because it is normalized by the area in which the star formation occurs, and this indirectly considers the gas density. The $Σ_{\rm SFR}$-$M_*$ diagram has been shown to complement the widely used (specific) SFR-$M_*$, particularly when selecting passive galaxies. We establish the $Σ_{\rm SFR}$\ evolution since $z=12$ in the framework of the L-Galaxies2020 semi-analytical model (SAM), and we interpret recent observations. We estimated $Σ_{\rm SFR}$(-$M_*$) and the cosmic star formation rate density (CSFRD) for the simulated galaxy population and for the subsamples, which were divided into stellar mass bins in the given redshift. The simulated $Σ_{\rm SFR}$\ decreases by $\sim3.5$ dex from $z=12$ to $z=0$. We show that galaxies with different stellar masses have different paths of $Σ_{\rm SFR}$\ evolution. We find that $Σ_{\rm SFR}{\rm MS}$\ is already observed at $z\sim11$. The simulated $Σ_{\rm SFR}{\rm MS}$\ agrees with the observed one at $z=0, 1, 2, 5$, and $7.5$ and with individual galaxies at $z>10$. We show that the highest $Σ_{\rm SFR}{\rm MS}$\ slope of $0.709\pm0.005$ is at $z\sim3$ and decreases to $\sim0.085\pm0.003$ at $z=0$. This is mostly driven by a rapid decrease in SFR with an additional size increase for the most massive galaxies in this redshift range. This coincides with the dominance of the most massive galaxies in the CSFRD from the SAM.
△ Less
Submitted 9 December, 2024; v1 submitted 29 November, 2024;
originally announced December 2024.
-
The Fate of the Interstellar Medium in Early-type Galaxies. V. AGN Feedback from Optical Spectral Classification
Authors:
Oleh Ryzhov,
Michał J. Michałowski,
J. Nadolny,
J. Hjorth,
A. Leśniewska,
M. Solar,
P. Nowaczyk,
C. Gall,
T. T. Takeuchi
Abstract:
Quenching of star-formation plays a fundamental role in galaxy evolution. This process occurs due to the removal of the cold interstellar medium (ISM) or stabilization against collapse, so that gas cannot be used in the formation of new stars. In this paper, we study the effect of different mechanisms of ISM removal. In particular, we revised the well-known Baldwin-Philips-Terlevich (BPT) and…
▽ More
Quenching of star-formation plays a fundamental role in galaxy evolution. This process occurs due to the removal of the cold interstellar medium (ISM) or stabilization against collapse, so that gas cannot be used in the formation of new stars. In this paper, we study the effect of different mechanisms of ISM removal. In particular, we revised the well-known Baldwin-Philips-Terlevich (BPT) and $\mathrm{EW_{Hα}}$ vs. $\mathrm{[NII]/Hα}$ (WHAN) emission-line ratio diagnostics, so that we could classify all galaxies, even those not detected at some emission lines, introducing several new spectral classes. We use spectroscopic data and several physical parameters of 2409 dusty early-type galaxies in order to find out the dominant ionization source [active galactic nuclei (AGNs), young massive stars, hot low-mass evolved stars (HOLMES)] and its effect on the ISM. We find that strong AGNs can play a significant role in the ISM removal process only for galaxies with ages lower than $10^{9.4}$ yr, but we cannot rule out the influence of weak AGNs at any age. For older galaxies, HOLMES/planetary nebulae contribute significantly to the ISM removal process. Additionally, we provide the BPT and WHAN classifications not only for the selected sample but also for all 300000 galaxies in the GAMA fields.
△ Less
Submitted 15 November, 2024;
originally announced November 2024.
-
Binary progenitor systems for Type Ic supernovae
Authors:
Martín Solar,
Michał J. Michałowski,
Jakub Nadolny,
Lluís Galbany,
Jens Hjorth,
Emmanouil Zapartas,
Jesper Sollerman,
Leslie Hunt,
Sylvio Klose,
Maciej Koprowski,
Aleksandra Leśniewska,
Michał Małkowski,
Ana M. Nicuesa Guelbenzu,
Oleh Ryzhov,
Sandra Savaglio,
Patricia Schady,
Steve Schulze,
Antonio de Ugarte Postigo,
Susanna D. Vergani,
Darach Watson,
Radosław Wróblewski
Abstract:
Core-collapse supernovae are explosions of massive stars at the end of their evolution. They are responsible for metal production and for halting star formation, having a significant impact on galaxy evolution. The details of these processes depend on the nature of supernova progenitors, but it is unclear if Type Ic supernovae (without hydrogen or helium lines in their spectra) originate from core…
▽ More
Core-collapse supernovae are explosions of massive stars at the end of their evolution. They are responsible for metal production and for halting star formation, having a significant impact on galaxy evolution. The details of these processes depend on the nature of supernova progenitors, but it is unclear if Type Ic supernovae (without hydrogen or helium lines in their spectra) originate from core-collapses of very massive stars (> 30 Msun) or from less massive stars in binary systems. Here we show that Type II (with hydrogen lines) and Ic supernovae are located in environments with similar molecular gas densities, therefore their progenitors have comparable lifetimes and initial masses. This supports a binary interaction for most Type Ic supernova progenitors, which explains the lack of hydrogen and helium lines. This finding can be implemented in sub-grid prescriptions in numerical cosmological simulations to improve the feedback and chemical mixing.
△ Less
Submitted 3 September, 2024;
originally announced September 2024.
-
The fate of the interstellar medium in early-type galaxies. IV. The impact of stellar feedback, mergers, and black holes on the cold ISM in simulated galaxies
Authors:
Jakub Nadolny,
Michał J. Michałowski,
Massimiliano Parente,
Jens Hjorth,
Christa Gall,
Aleksandra Leśniewska,
Martín Solar,
Przemysław Nowaczyk,
Oleh Ryzhov
Abstract:
Removing cold interstellar medium (ISM) from a galaxy is central to quenching star formation. However, the exact mechanism of this process remains unclear. The objective of this work is to find the mechanism responsible for dust and gas removal in simulated early-type galaxies (ETGs). A statistically significant sample of massive (M_*>$10^{10}$M$_\odot$), simulated ETG in a redshift range of 0.02-…
▽ More
Removing cold interstellar medium (ISM) from a galaxy is central to quenching star formation. However, the exact mechanism of this process remains unclear. The objective of this work is to find the mechanism responsible for dust and gas removal in simulated early-type galaxies (ETGs). A statistically significant sample of massive (M_*>$10^{10}$M$_\odot$), simulated ETG in a redshift range of 0.02--0.32 is studied in the context of its ISM properties. In particular, we investigate the cold dust and gas removal timescales, the cold gas inflows, and their relation with black hole (BH) mass. We also investigate the evolution of galaxies in the dust vs. star formation rate (SFR) plane and the influence of merger events. We find agreement with previous observational works considering the timescales of dust and HI removal from ETGs. When considering the dust-to-stellar mass ratio as a function of time in simulations, we recovered a similar decline as in the observational sample as a function of stellar age, validating its use for timing the ISM decline. Moreover, we recover the observed relation between dust mass and SFR for actively star-forming galaxies as well as for passive ETGs. We also show that starburst galaxies form their own sequence on the dust vs. SFR plot in a form $\log(M_{\rm dust, SB})= 0.913\times \log({\rm SFR}) + 6.533$ with $2σ$ scatter of 0.32. Finally, we find that type II supernova reverse shocks dominate the dust destruction at the early stages of ETG evolution, while at later times stellar feedback becomes more important. We show that merger events lead to morphological transformations by increasing the bulge-to-total stellar mass ratio followed by an increase in BH masses. The BH feedback resulting from radio mode accretion prevents the hot halo gas from cooling, indirectly leading to a decrease in the SFR.
△ Less
Submitted 27 June, 2024; v1 submitted 24 June, 2024;
originally announced June 2024.
-
The fate of the interstellar medium in early-type galaxies. III. The mechanism of ISM removal and quenching of star formation
Authors:
Michał J. Michałowski,
C. Gall,
J. Hjorth,
D. T. Frayer,
A. -L. Tsai,
K. Rowlands,
T. T. Takeuchi,
A. Leśniewska,
D. Behrendt,
N. Bourne,
D. H. Hughes,
M. P. Koprowski,
J. Nadolny,
O. Ryzhov,
M. Solar,
E. Spring,
J. Zavala,
P. Bartczak
Abstract:
Understanding how galaxies quench their star formation is crucial for studies of galaxy evolution. Quenching is related to the cold gas decrease. In the first paper we showed that the dust removal timescale in early-type galaxies (ETGs) is about 2.5 Gyr. Here we present carbon monoxide (CO) and 21 cm hydrogen (H I) line observations of these galaxies and measure the timescale of removal of the col…
▽ More
Understanding how galaxies quench their star formation is crucial for studies of galaxy evolution. Quenching is related to the cold gas decrease. In the first paper we showed that the dust removal timescale in early-type galaxies (ETGs) is about 2.5 Gyr. Here we present carbon monoxide (CO) and 21 cm hydrogen (H I) line observations of these galaxies and measure the timescale of removal of the cold interstellar medium (ISM). We find that all the cold ISM components (dust, molecular and atomic gas) decline at similar rates. This allows us to rule out a wide range of potential ISM removal mechanisms (including starburst-driven outflows, astration, a decline in the number of asymptotic giant branch stars), and artificial effects like stellar mass-age correlation, environmental influence, mergers, and selection bias, leaving ionization by evolved low-mass stars and ionization/outflows by supernovae Type Ia or active galactic nuclei as viable mechanisms. We also provide evidence for an internal origin of the detected ISM. Moreover, we find that the quenching of star formation in these galaxies cannot be explained by a reduction in gas amount alone, because the star formation rates (SFRs) decrease faster (on a timescale of about 1.8 Gyr) than the amount of cold gas. Furthermore, the star formation efficiency of the ETGs (SFE = SFR/MH2) is lower than that of star-forming galaxies, whereas their gas mass fractions (fH2 = MH2/M*) are normal. This may be explained by the stabilization of gas against fragmentation, for example due to morphological quenching, turbulence, or magnetic fields.
△ Less
Submitted 26 March, 2024; v1 submitted 9 January, 2024;
originally announced January 2024.
-
Semidiscrete optimal transport with unknown costs
Authors:
Yinchu Zhu,
Ilya O. Ryzhov
Abstract:
Semidiscrete optimal transport is a challenging generalization of the classical transportation problem in linear programming. The goal is to design a joint distribution for two random variables (one continuous, one discrete) with fixed marginals, in a way that minimizes expected cost. We formulate a novel variant of this problem in which the cost functions are unknown, but can be learned through n…
▽ More
Semidiscrete optimal transport is a challenging generalization of the classical transportation problem in linear programming. The goal is to design a joint distribution for two random variables (one continuous, one discrete) with fixed marginals, in a way that minimizes expected cost. We formulate a novel variant of this problem in which the cost functions are unknown, but can be learned through noisy observations; however, only one function can be sampled at a time. We develop a semi-myopic algorithm that couples online learning with stochastic approximation, and prove that it achieves optimal convergence rates, despite the non-smoothness of the stochastic gradient and the lack of strong concavity in the objective function.
△ Less
Submitted 28 February, 2025; v1 submitted 1 October, 2023;
originally announced October 2023.
-
The Fate of the Interstellar Medium in Early-type Galaxies. II. Observational Evidence for Morphological Quenching
Authors:
Aleksandra Leśniewska,
Michał Jerzy Michałowski,
Christa Gall,
Jens Hjorth,
Jakub Nadolny,
Oleh Ryzhov,
Martin Solar
Abstract:
The mechanism by which galaxies stop forming stars and get rid of their interstellar medium (ISM) remains elusive. Here, we study a sample of more than two thousand elliptical galaxies in which dust emission has been detected. This is the largest sample of such galaxies ever analysed. We infer the timescale for removal of dust in these galaxies and investigate its dependency on physical and enviro…
▽ More
The mechanism by which galaxies stop forming stars and get rid of their interstellar medium (ISM) remains elusive. Here, we study a sample of more than two thousand elliptical galaxies in which dust emission has been detected. This is the largest sample of such galaxies ever analysed. We infer the timescale for removal of dust in these galaxies and investigate its dependency on physical and environmental properties. We obtain a dust removal timescale in elliptical galaxies of $τ$ = 2.26 $\pm$ 0.18 Gyr, corresponding to a half-life time of 1.57 $\pm$ 0.12 Gyr. This timescale does not depend on environment, stellar mass or redshift. We observe a departure of dusty elliptical galaxies from the star formation rate vs. dust mass relation. This is caused by the star-formation rates declining faster than the dust masses and indicates that there exists an internal mechanism, which affects star formation, but leaves the ISM intact. Morphological quenching together with ionisation or outflows caused by older stellar populations (supernova type Ia or planetary nebulae) are consistent with these observations.
△ Less
Submitted 9 June, 2023;
originally announced June 2023.
-
Optimal data-driven hiring with equity for underrepresented groups
Authors:
Yinchu Zhu,
Ilya O. Ryzhov
Abstract:
We present a data-driven prescriptive framework for fair decisions, motivated by hiring. An employer evaluates a set of applicants based on their observable attributes. The goal is to hire the best candidates while avoiding bias with regard to a certain protected attribute. Simply ignoring the protected attribute will not eliminate bias due to correlations in the data. We present a hiring policy t…
▽ More
We present a data-driven prescriptive framework for fair decisions, motivated by hiring. An employer evaluates a set of applicants based on their observable attributes. The goal is to hire the best candidates while avoiding bias with regard to a certain protected attribute. Simply ignoring the protected attribute will not eliminate bias due to correlations in the data. We present a hiring policy that depends on the protected attribute functionally, but not statistically, and we prove that, among all possible fair policies, ours is optimal with respect to the firm's objective. We test our approach on both synthetic and real data, and find that it shows great practical potential to improve equity for underrepresented and historically marginalized groups.
△ Less
Submitted 18 June, 2022;
originally announced June 2022.
-
Policy Optimization in Dynamic Bayesian Network Hybrid Models of Biomanufacturing Processes
Authors:
Hua Zheng,
Wei Xie,
Ilya O. Ryzhov,
Dongming Xie
Abstract:
Biopharmaceutical manufacturing is a rapidly growing industry with impact in virtually all branches of medicines. Biomanufacturing processes require close monitoring and control, in the presence of complex bioprocess dynamics with many interdependent factors, as well as extremely limited data due to the high cost of experiments as well as the novelty of personalized bio-drugs. We develop a novel m…
▽ More
Biopharmaceutical manufacturing is a rapidly growing industry with impact in virtually all branches of medicines. Biomanufacturing processes require close monitoring and control, in the presence of complex bioprocess dynamics with many interdependent factors, as well as extremely limited data due to the high cost of experiments as well as the novelty of personalized bio-drugs. We develop a novel model-based reinforcement learning framework that can achieve human-level control in low-data environments. The model uses a dynamic Bayesian network to capture causal interdependencies between factors and predict how the effects of different inputs propagate through the pathways of the bioprocess mechanisms. This enables the design of process control policies that are both interpretable and robust against model risk. We present a computationally efficient, provably convergence stochastic gradient method for optimizing such policies. Validation is conducted on a realistic application with a multi-dimensional, continuous state variable.
△ Less
Submitted 24 July, 2022; v1 submitted 13 May, 2021;
originally announced May 2021.
-
Personalized Multimorbidity Management for Patients with Type 2 Diabetes Using Reinforcement Learning of Electronic Health Records
Authors:
Hua Zheng,
Ilya O. Ryzhov,
Wei Xie,
Judy Zhong
Abstract:
Comorbid chronic conditions are common among people with type 2 diabetes. We developed an Artificial Intelligence algorithm, based on Reinforcement Learning (RL), for personalized diabetes and multi-morbidity management with strong potential to improve health outcomes relative to current clinical practice. In this paper, we modeled glycemia, blood pressure and cardiovascular disease (CVD) risk as…
▽ More
Comorbid chronic conditions are common among people with type 2 diabetes. We developed an Artificial Intelligence algorithm, based on Reinforcement Learning (RL), for personalized diabetes and multi-morbidity management with strong potential to improve health outcomes relative to current clinical practice. In this paper, we modeled glycemia, blood pressure and cardiovascular disease (CVD) risk as health outcomes using a retrospective cohort of 16,665 patients with type 2 diabetes from New York University Langone Health ambulatory care electronic health records in 2009 to 2017. We trained a RL prescription algorithm that recommends a treatment regimen optimizing patients' cumulative health outcomes using their individual characteristics and medical history at each encounter. The RL recommendations were evaluated on an independent subset of patients. The results demonstrate that the proposed personalized reinforcement learning prescriptive framework for type 2 diabetes yielded high concordance with clinicians' prescriptions and substantial improvements in glycemia, blood pressure, cardiovascular disease risk outcomes.
△ Less
Submitted 28 October, 2020;
originally announced November 2020.
-
A New Optimal Stepsize For Approximate Dynamic Programming
Authors:
Ilya O. Ryzhov,
Peter I. Frazier,
Warren B. Powell
Abstract:
Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value function approximation. Many operations research applications are computationally inte…
▽ More
Approximate dynamic programming (ADP) has proven itself in a wide range of applications spanning large-scale transportation problems, health care, revenue management, and energy systems. The design of effective ADP algorithms has many dimensions, but one crucial factor is the stepsize rule used to update a value function approximation. Many operations research applications are computationally intensive, and it is important to obtain good results quickly. Furthermore, the most popular stepsize formulas use tunable parameters and can produce very poor results if tuned improperly. We derive a new stepsize rule that optimizes the prediction error in order to improve the short-term performance of an ADP algorithm. With only one, relatively insensitive tunable parameter, the new rule adapts to the level of noise in the problem and produces faster convergence in numerical experiments.
△ Less
Submitted 13 July, 2014; v1 submitted 9 July, 2014;
originally announced July 2014.
