Showing 1–50 of 11,940 results for author: N

Nonparametric Bayesian analysis for the Galton-Watson process

Authors: Massimo Cannas, Michele Guindani, Nicola Piras

Abstract: The Galton-Watson process is a model for population growth which assumes that individuals reproduce independently according to the same offspring distribution. Inference usually focuses on the offspring average as it allows to classify the process with respect to extinction. We propose a fully non-parametric approach for Bayesian inference on the GW model using a Dirichlet Process prior. The prior… ▽ More The Galton-Watson process is a model for population growth which assumes that individuals reproduce independently according to the same offspring distribution. Inference usually focuses on the offspring average as it allows to classify the process with respect to extinction. We propose a fully non-parametric approach for Bayesian inference on the GW model using a Dirichlet Process prior. The prior naturally generalizes the Dirichlet conjugate prior distribution, and it allows learning the support of the offspring distribution from the data as well as taking into account possible overdispersion of the data. The performance of the proposed approach is compared with both frequentist and Bayesian procedures via simulation. In particular, we show that the use of a DP prior yields good classification performance with both complete and incomplete data. A real-world data example concerning COVID-19 data from Sardinia illustrates the use of the approach in practice. △ Less

Submitted 26 June, 2025; originally announced June 2025.

Comments: 27 pages, 2 figures

MSC Class: 62G05; 62F15

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

Authors: Soufiane Hayou, Nikhil Ghosh, Bin Yu

Abstract: Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of full finetuning. Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter pla… ▽ More Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models. Its small memory footprint allows practitioners to adapt large models to specific tasks at a fraction of the cost of full finetuning. Different modifications have been proposed to enhance its efficiency by, for example, setting the learning rate, the rank, and the initialization. Another improvement axis is adapter placement strategy: when using LoRA, practitioners usually pick module types to adapt with LoRA, such as Query and Key modules. Few works have studied the problem of adapter placement, with nonconclusive results: original LoRA paper suggested placing adapters in attention modules, while other works suggested placing them in the MLP modules. Through an intuitive theoretical analysis, we introduce PLoP (Precise LoRA Placement), a lightweight method that allows automatic identification of module types where LoRA adapters should be placed, given a pretrained model and a finetuning task. We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies through comprehensive experiments on supervised finetuning and reinforcement learning for reasoning. △ Less

Submitted 25 June, 2025; originally announced June 2025.

Comments: TD,LR: A lightweight module type selection method for LoRA finetuning. PLoP gives precise placements for LoRA adapters for improved performance

LARP: Learner-Agnostic Robust Data Prefiltering

Authors: Kristian Minchev, Dimitar Iliev Dimitrov, Nikola Konstantinov

Abstract: The widespread availability of large public datasets is a key factor behind the recent successes of statistical inference and machine learning methods. However, these datasets often contain some low-quality or contaminated data, to which many learning procedures are sensitive. Therefore, the question of whether and how public datasets should be prefiltered to facilitate accurate downstream learnin… ▽ More The widespread availability of large public datasets is a key factor behind the recent successes of statistical inference and machine learning methods. However, these datasets often contain some low-quality or contaminated data, to which many learning procedures are sensitive. Therefore, the question of whether and how public datasets should be prefiltered to facilitate accurate downstream learning arises. On a technical level this requires the construction of principled data prefiltering methods which are learner-agnostic robust, in the sense of provably protecting a set of pre-specified downstream learners from corrupted data. In this work, we formalize the problem of Learner-Agnostic Robust data Prefiltering (LARP), which aims at finding prefiltering procedures that minimize a worst-case loss over a pre-specified set of learners. We first instantiate our framework in the context of scalar mean estimation with Huber estimators under the Huber data contamination model. We provide a hardness result on a specific problem instance and analyze several natural prefiltering procedures. Our theoretical results indicate that performing LARP on a heterogeneous set of learners leads to some loss in model performance compared to the alternative of prefiltering data for each learner/use-case individually. We explore the resulting utility loss and its dependence on the problem parameters via extensive experiments on real-world image and tabular data, observing statistically significant reduction in utility. Finally, we model the trade-off between the utility drop and the cost of repeated (learner-specific) prefiltering within a game-theoretic framework and showcase benefits of LARP for large datasets. △ Less

Submitted 25 June, 2025; originally announced June 2025.

Path-Based Approach for Detecting and Assessing Inconsistency in Network Meta-Analysis: A Novel Method

Authors: Noosheen R. Tahmasebi, Annabel L. Davies, Theodoros Papakonstantinou, Gerta Rücker, Adriani Nikolakopoulou

Abstract: Network Meta-Analysis (NMA) plays a pivotal role in synthesizing evidence from various sources and comparing multiple interventions. At its core, NMA relies on integrating both direct evidence from head-to-head comparisons and indirect evidence from different paths that link treatments through common comparators. A key aspect is evaluating consistency between direct and indirect sources. Existing… ▽ More Network Meta-Analysis (NMA) plays a pivotal role in synthesizing evidence from various sources and comparing multiple interventions. At its core, NMA relies on integrating both direct evidence from head-to-head comparisons and indirect evidence from different paths that link treatments through common comparators. A key aspect is evaluating consistency between direct and indirect sources. Existing methods to detect inconsistency, although widely used, have limitations. For example, they do not account for differences within indirect sources or cannot estimate inconsistency when direct evidence is absent. In this paper, we introduce a path-based approach that explores all sources of evidence without separating direct and indirect. We introduce a measure based on the square of differences to quantitatively capture inconsistency, and propose a Netpath plot to visualize inconsistencies between various paths. We provide an implementation of our path-based method within the netmeta R package. Via application to fictional and real-world examples, we show that our method is able to detect and visualize inconsistency between multiple paths of evidence that would otherwise be masked by considering all indirect sources together. The path-based approach therefore provides a more comprehensive evaluation of inconsistency within a network of treatments. △ Less

Submitted 25 June, 2025; originally announced June 2025.

Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining

Authors: Nathan Stromberg, Christos Thrampoulidis, Lalitha Sankar

Abstract: While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized set… ▽ More While machine learning models become more capable in discriminative tasks at scale, their ability to overcome biases introduced by training data has come under increasing scrutiny. Previous results suggest that there are two extremes of parameterization with very different behaviors: the population (underparameterized) setting where loss weighting is optimal and the separable overparameterized setting where loss weighting is ineffective at ensuring equal performance across classes. This work explores the regime of last layer retraining (LLR) in which the unseen limited (retraining) data is frequently inseparable and the model proportionately sized, falling between the two aforementioned extremes. We show, in theory and practice, that loss weighting is still effective in this regime, but that these weights \emph{must} take into account the relative overparameterization of the model. △ Less

Submitted 24 June, 2025; originally announced June 2025.

Elucidated Rolling Diffusion Models for Probabilistic Weather Forecasting

Authors: Salva Rühling Cachay, Miika Aittala, Karsten Kreis, Noah Brenowitz, Arash Vahdat, Morteza Mardani, Rose Yu

Abstract: Diffusion models are a powerful tool for probabilistic forecasting, yet most applications in high-dimensional chaotic systems predict future snapshots one-by-one. This common approach struggles to model complex temporal dependencies and fails to explicitly account for the progressive growth of uncertainty inherent to such systems. While rolling diffusion frameworks, which apply increasing noise to… ▽ More Diffusion models are a powerful tool for probabilistic forecasting, yet most applications in high-dimensional chaotic systems predict future snapshots one-by-one. This common approach struggles to model complex temporal dependencies and fails to explicitly account for the progressive growth of uncertainty inherent to such systems. While rolling diffusion frameworks, which apply increasing noise to forecasts at longer lead times, have been proposed to address this, their integration with state-of-the-art, high-fidelity diffusion techniques remains a significant challenge. We tackle this problem by introducing Elucidated Rolling Diffusion Models (ERDM), the first framework to successfully unify a rolling forecast structure with the principled, performant design of Elucidated Diffusion Models (EDM). To do this, we adapt the core EDM components-its noise schedule, network preconditioning, and Heun sampler-to the rolling forecast setting. The success of this integration is driven by three key contributions: (i) a novel loss weighting scheme that focuses model capacity on the mid-range forecast horizons where determinism gives way to stochasticity; (ii) an efficient initialization strategy using a pre-trained EDM for the initial window; and (iii) a bespoke hybrid sequence architecture for robust spatiotemporal feature extraction under progressive denoising. On 2D Navier-Stokes simulations and ERA5 global weather forecasting at 1.5^\circ resolution, ERDM consistently outperforms key diffusion-based baselines, including conditional autoregressive EDM. ERDM offers a flexible and powerful general framework for tackling diffusion-based sequence generation problems where modeling escalating uncertainty is paramount. Code is available at: https://github.com/salvaRC/erdm △ Less

Submitted 24 June, 2025; originally announced June 2025.

An ab initio foundation model of wavefunctions that accurately describes chemical bond breaking

Authors: Adam Foster, Zeno Schätzle, P. Bernát Szabó, Lixue Cheng, Jonas Köhler, Gino Cassella, Nicholas Gao, Jiawei Li, Frank Noé, Jan Hermann

Abstract: Reliable description of bond breaking remains a major challenge for quantum chemistry due to the multireferential character of the electronic structure in dissociating species. Multireferential methods in particular suffer from large computational cost, which under the normal paradigm has to be paid anew for each system at a full price, ignoring commonalities in electronic structure across molecul… ▽ More Reliable description of bond breaking remains a major challenge for quantum chemistry due to the multireferential character of the electronic structure in dissociating species. Multireferential methods in particular suffer from large computational cost, which under the normal paradigm has to be paid anew for each system at a full price, ignoring commonalities in electronic structure across molecules. Quantum Monte Carlo with deep neural networks (deep QMC) uniquely offers to exploit such commonalities by pretraining transferable wavefunction models, but all such attempts were so far limited in scope. Here, we bring this new paradigm to fruition with Orbformer, a novel transferable wavefunction model pretrained on 22,000 equilibrium and dissociating structures that can be fine-tuned on unseen molecules reaching an accuracy-cost ratio rivalling classical multireferential methods. On established benchmarks as well as more challenging bond dissociations and Diels-Alder reactions, Orbformer is the only method that consistently converges to chemical accuracy (1 kcal/mol). This work turns the idea of amortizing the cost of solving the Schrödinger equation over many molecules into a practical approach in quantum chemistry. △ Less

Submitted 24 June, 2025; originally announced June 2025.

Cross-regularization: Adaptive Model Complexity through Validation Gradients

Authors: Carlos Stein Brito

Abstract: Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-valid… ▽ More Model regularization requires extensive manual tuning to balance complexity against overfitting. Cross-regularization resolves this tradeoff by directly adapting regularization parameters through validation gradients during training. The method splits parameter optimization - training data guides feature learning while validation data shapes complexity controls - converging provably to cross-validation optima. When implemented through noise injection in neural networks, this approach reveals striking patterns: unexpectedly high noise tolerance and architecture-specific regularization that emerges organically during training. Beyond complexity control, the framework integrates seamlessly with data augmentation, uncertainty calibration and growing datasets while maintaining single-run efficiency through a simple gradient-based approach. △ Less

Submitted 24 June, 2025; originally announced June 2025.

Comments: 21 pages, 13 figures. Accepted at ICML 2025

Geometric-Aware Variational Inference: Robust and Adaptive Regularization with Directional Weight Uncertainty

Authors: Carlos Stein Brito

Abstract: Deep neural networks require principled uncertainty quantification, yet existing variational inference methods often employ isotropic Gaussian approximations in weight space that poorly match the network's inherent geometry. We address this mismatch by introducing Concentration-Adapted Perturbations (CAP), a variational framework that models weight uncertainties directly on the unit hypersphere us… ▽ More Deep neural networks require principled uncertainty quantification, yet existing variational inference methods often employ isotropic Gaussian approximations in weight space that poorly match the network's inherent geometry. We address this mismatch by introducing Concentration-Adapted Perturbations (CAP), a variational framework that models weight uncertainties directly on the unit hypersphere using von Mises-Fisher distributions. Building on recent work in radial-directional posterior decompositions and spherical weight constraints, CAP provides the first complete theoretical framework connecting directional statistics to practical noise regularization in neural networks. Our key contribution is an analytical derivation linking vMF concentration parameters to activation noise variance, enabling each layer to learn its optimal uncertainty level through a novel closed-form KL divergence regularizer. In experiments on CIFAR-10, CAP significantly improves model calibration - reducing Expected Calibration Error by 5.6x - while providing interpretable layer-wise uncertainty profiles. CAP requires minimal computational overhead and integrates seamlessly into standard architectures, offering a theoretically grounded yet practical approach to uncertainty quantification in deep learning. △ Less

Submitted 24 June, 2025; originally announced June 2025.

Comments: 19 pages, 4 figures

Iterative Quantum Feature Maps

Authors: Nasa Matsumoto, Quoc Hoan Tran, Koki Chinzei, Yasuhiro Endo, Hirotaka Oshima

Abstract: Quantum machine learning models that leverage quantum circuits as quantum feature maps (QFMs) are recognized for their enhanced expressive power in learning tasks. Such models have demonstrated rigorous end-to-end quantum speedups for specific families of classification problems. However, deploying deep QFMs on real quantum hardware remains challenging due to circuit noise and hardware constraints… ▽ More Quantum machine learning models that leverage quantum circuits as quantum feature maps (QFMs) are recognized for their enhanced expressive power in learning tasks. Such models have demonstrated rigorous end-to-end quantum speedups for specific families of classification problems. However, deploying deep QFMs on real quantum hardware remains challenging due to circuit noise and hardware constraints. Additionally, variational quantum algorithms often suffer from computational bottlenecks, particularly in accurate gradient estimation, which significantly increases quantum resource demands during training. We propose Iterative Quantum Feature Maps (IQFMs), a hybrid quantum-classical framework that constructs a deep architecture by iteratively connecting shallow QFMs with classically computed augmentation weights. By incorporating contrastive learning and a layer-wise training mechanism, IQFMs effectively reduces quantum runtime and mitigates noise-induced degradation. In tasks involving noisy quantum data, numerical experiments show that IQFMs outperforms quantum convolutional neural networks, without requiring the optimization of variational quantum parameters. Even for a typical classical image classification benchmark, a carefully designed IQFMs achieves performance comparable to that of classical neural networks. This framework presents a promising path to address current limitations and harness the full potential of quantum-enhanced machine learning. △ Less

Submitted 24 June, 2025; originally announced June 2025.

Comments: 13 pages, 12 figures

Unlocking Insights Addressing Alcohol Inference Mismatch through Database-Narrative Alignment

Authors: Sudesh Bhagat, Raghupathi Kandiboina, Ibne Farabi Shihab, Skylar Knickerbocker, Neal Hawkins, Anuj Sharma

Abstract: Road traffic crashes are a significant global cause of fatalities, emphasizing the urgent need for accurate crash data to enhance prevention strategies and inform policy development. This study addresses the challenge of alcohol inference mismatch (AIM) by employing database narrative alignment to identify AIM in crash data. A framework was developed to improve data quality in crash management sys… ▽ More Road traffic crashes are a significant global cause of fatalities, emphasizing the urgent need for accurate crash data to enhance prevention strategies and inform policy development. This study addresses the challenge of alcohol inference mismatch (AIM) by employing database narrative alignment to identify AIM in crash data. A framework was developed to improve data quality in crash management systems and reduce the percentage of AIM crashes. Utilizing the BERT model, the analysis of 371,062 crash records from Iowa (2016-2022) revealed 2,767 AIM incidents, resulting in an overall AIM percentage of 24.03%. Statistical tools, including the Probit Logit model, were used to explore the crash characteristics affecting AIM patterns. The findings indicate that alcohol-related fatal crashes and nighttime incidents have a lower percentage of the mismatch, while crashes involving unknown vehicle types and older drivers are more susceptible to mismatch. The geospatial cluster as part of this study can identify the regions which have an increased need for education and training. These insights highlight the necessity for targeted training programs and data management teams to improve the accuracy of crash reporting and support evidence-based policymaking. △ Less

Submitted 24 June, 2025; originally announced June 2025.

When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Nets

Authors: Chen Zeno, Hila Manor, Greg Ongie, Nir Weinberger, Tomer Michaeli, Daniel Soudry

Abstract: While diffusion models generate high-quality images via probability flow, the theoretical understanding of this process remains incomplete. A key question is when probability flow converges to training samples or more general points on the data manifold. We analyze this by studying the probability flow of shallow ReLU neural network denoisers trained with minimal $\ell^2$ norm. For intuition, we i… ▽ More While diffusion models generate high-quality images via probability flow, the theoretical understanding of this process remains incomplete. A key question is when probability flow converges to training samples or more general points on the data manifold. We analyze this by studying the probability flow of shallow ReLU neural network denoisers trained with minimal $\ell^2$ norm. For intuition, we introduce a simpler score flow and show that for orthogonal datasets, both flows follow similar trajectories, converging to a training point or a sum of training points. However, early stopping by the diffusion time scheduler allows probability flow to reach more general manifold points. This reflects the tendency of diffusion models to both memorize training samples and generate novel points that combine aspects of multiple samples, motivating our study of such behavior in simplified settings. We extend these results to obtuse simplex data and, through simulations in the orthogonal case, confirm that probability flow converges to a training point, a sum of training points, or a manifold point. Moreover, memorization decreases when the number of training samples grows, as fewer samples accumulate near training points. △ Less

Submitted 23 June, 2025; originally announced June 2025.

Comments: Accepted to the Forty-second International Conference on Machine Learning (ICML 2025)

A Practical Introduction to Regression-based Causal Inference in Meteorology (I): All confounders measured

Authors: Caren Marzban, Yikun Zhang, Nicholas Bond, Michael Richman

Abstract: Whether a variable is the cause of another, or simply associated with it, is often an important scientific question. Causal Inference is the name associated with the body of techniques for addressing that question in a statistical setting. Although assessing causality is relatively straightforward in the presence of temporal information, outside of that setting - the situation considered here - it… ▽ More Whether a variable is the cause of another, or simply associated with it, is often an important scientific question. Causal Inference is the name associated with the body of techniques for addressing that question in a statistical setting. Although assessing causality is relatively straightforward in the presence of temporal information, outside of that setting - the situation considered here - it is more difficult to assess causal effects. The development of the field of causal inference has involved concepts from a wide range of topics, thereby limiting its adoption across some fields, including meteorology. However, at its core, the requisite knowledge for causal inference involves little more than basic probability theory and regression, topics familiar to most meteorologists. By focusing on these core areas, this and a companion article provide a steppingstone for the meteorology community into the field of (non-temporal) causal inference. Although some theoretical foundations are presented, the main goal is the application of a specific method, called matching, to a problem in meteorology. The data for the application are in public domain, and R code is provided as well, forming an easy path for meteorology students and researchers to enter the field. △ Less

Submitted 24 June, 2025; v1 submitted 23 June, 2025; originally announced June 2025.

The Within-Orbit Adaptive Leapfrog No-U-Turn Sampler

Authors: Nawaf Bou-Rabee, Bob Carpenter, Tore Selland Kleppe, Sifan Liu

Abstract: Locally adapting parameters within Markov chain Monte Carlo methods while preserving reversibility is notoriously difficult. The success of the No-U-Turn Sampler (NUTS) largely stems from its clever local adaptation of the integration time in Hamiltonian Monte Carlo via a geometric U-turn condition. However, posterior distributions frequently exhibit multi-scale geometries with extreme variations… ▽ More Locally adapting parameters within Markov chain Monte Carlo methods while preserving reversibility is notoriously difficult. The success of the No-U-Turn Sampler (NUTS) largely stems from its clever local adaptation of the integration time in Hamiltonian Monte Carlo via a geometric U-turn condition. However, posterior distributions frequently exhibit multi-scale geometries with extreme variations in scale, making it necessary to also adapt the leapfrog integrator's step size locally and dynamically. Despite its practical importance, this problem has remained largely open since the introduction of NUTS by Hoffman and Gelman (2014). To address this issue, we introduce the Within-orbit Adaptive Leapfrog No-U-Turn Sampler (WALNUTS), a generalization of NUTS that adapts the leapfrog step size at fixed intervals of simulated time as the orbit evolves. At each interval, the algorithm selects the largest step size from a dyadic schedule that keeps the energy error below a user-specified threshold. Like NUTS, WALNUTS employs biased progressive state selection to favor states with positions that are further from the initial point along the orbit. Empirical evaluations on multiscale target distributions, including Neal's funnel and the Stock-Watson stochastic volatility time-series model, demonstrate that WALNUTS achieves substantial improvements in sampling efficiency and robustness compared to standard NUTS. △ Less

Submitted 23 June, 2025; originally announced June 2025.

Comments: for companion GitHub repo, see https://github.com/bob-carpenter/walnuts

100-Day Analysis of USD/IDR Exchange Rate Dynamics Around the 2025 U.S. Presidential Inauguration

Authors: Sandy H. S. Herho, Siti N. Kaban, Cahya Nugraha

Abstract: Using a 100-day symmetric window around the January 2025 U.S. presidential inauguration, non-parametric statistical methods with bootstrap resampling (10,000 iterations) analyze distributional properties and anomalies. Results indicate a statistically significant 3.61\% Indonesian rupiah depreciation post-inauguration, with a large effect size (Cliff's Delta $= -0.9224$, CI: $[-0.9727, -0.8571]$).… ▽ More Using a 100-day symmetric window around the January 2025 U.S. presidential inauguration, non-parametric statistical methods with bootstrap resampling (10,000 iterations) analyze distributional properties and anomalies. Results indicate a statistically significant 3.61\% Indonesian rupiah depreciation post-inauguration, with a large effect size (Cliff's Delta $= -0.9224$, CI: $[-0.9727, -0.8571]$). Central tendency shifted markedly, yet volatility remained stable (variance ratio $= 0.9061$, $p = 0.504$). Four significant anomalies exhibiting temporal clustering are detected. These findings provide quantitative evidence of political transition effects on emerging market currencies, highlighting implications for monetary policy and currency risk management. △ Less

Submitted 23 June, 2025; originally announced June 2025.

Comments: 16 pages, 4 figures

A Structural Causal Model for Electronic Device Reliability: From Effects to Counterfactuals

Authors: Federico Mattia Stefanini, Nedka Dechkova Nikiforova, Rossella Berni

Abstract: Electronic devices exhibit changes in electrical resistance over time at varying rates, depending on the configuration of certain components. Since measuring overall electrical resistance requires partial disassembly, only a limited number of measurements are performed over thousands of operating hours. This leads to censored failure times, whether under natural stress or under accelerated stress… ▽ More Electronic devices exhibit changes in electrical resistance over time at varying rates, depending on the configuration of certain components. Since measuring overall electrical resistance requires partial disassembly, only a limited number of measurements are performed over thousands of operating hours. This leads to censored failure times, whether under natural stress or under accelerated stress conditions. To address these challenges, including device-specific failure thresholds, a parametric structural causal model is developed to extract information from both observational and experimental data, with the aim of estimating causal effects and counterfactuals, regardless of the applied stress regime. Synthetic data are used to illustrate the main findings. △ Less

Submitted 23 June, 2025; originally announced June 2025.

Comments: 23 pages, 7 figures

MSC Class: 62D20

A Practical Introduction to Regression-based Causal Inference in Meteorology (II): Unmeasured confounders

Authors: Caren Marzban, Yikun Zhang, Nicholas Bond, Michael Richman

Abstract: One obstacle to ``elevating" correlation to causation is the phenomenon of confounding, i.e., when a correlation between two variables exists because both variables are in fact caused by a third variable. The situation where the confounders are measured is examined in an earlier, accompanying article. Here, it is shown that even when the confounding variables are not measured, it is still possible… ▽ More One obstacle to ``elevating" correlation to causation is the phenomenon of confounding, i.e., when a correlation between two variables exists because both variables are in fact caused by a third variable. The situation where the confounders are measured is examined in an earlier, accompanying article. Here, it is shown that even when the confounding variables are not measured, it is still possible to estimate the causal effect via a regression-based method that uses the notion of Instrumental Variables. Using meteorological data set, similar to that in the sister article, a number of different estimates of the causal effect are compared and contrasted. It is shown that the Instrumental Variable results based on unmeasured confounders are consistent with those of the sister article where confounders are measured. △ Less

Submitted 23 June, 2025; originally announced June 2025.

Generalization under Byzantine & Poisoning Attacks: Tight Stability Bounds in Robust Distributed Learning

Authors: Thomas Boudou, Batiste Le Bars, Nirupam Gupta, Aurélien Bellet

Abstract: Robust distributed learning algorithms aim to maintain good performance in distributed and federated settings, even in the presence of misbehaving workers. Two primary threat models have been studied: Byzantine attacks, where misbehaving workers can send arbitrarily corrupted updates, and data poisoning attacks, where misbehavior is limited to manipulation of local training data. While prior work… ▽ More Robust distributed learning algorithms aim to maintain good performance in distributed and federated settings, even in the presence of misbehaving workers. Two primary threat models have been studied: Byzantine attacks, where misbehaving workers can send arbitrarily corrupted updates, and data poisoning attacks, where misbehavior is limited to manipulation of local training data. While prior work has shown comparable optimization error under both threat models, a fundamental question remains open: How do these threat models impact generalization? Empirical evidence suggests a gap between the two threat models, yet it remains unclear whether it is fundamental or merely an artifact of suboptimal attacks. In this work, we present the first theoretical investigation into this problem, formally showing that Byzantine attacks are intrinsically more harmful to generalization than data poisoning. Specifically, we prove that: (i) under data poisoning, the uniform algorithmic stability of a robust distributed learning algorithm, with optimal optimization error, degrades by an additive factor of $\varTheta ( \frac{f}{n-f} )$, with $f$ the number of misbehaving workers out of $n$; and (ii) In contrast, under Byzantine attacks, the degradation is in $\mathcal{O} \big( \sqrt{ \frac{f}{n-2f}} \big)$.This difference in stability leads to a generalization error gap that is especially significant as $f$ approaches its maximum value $\frac{n}{2}$. △ Less

Submitted 22 June, 2025; originally announced June 2025.

Triadic Novelty: A Typology and Measurement Framework for Recognizing Novel Contributions in Science

Authors: Jin Ai, Richard S. Steinberg, Chao Guo, Filipi Nascimento Silva

Abstract: Scientific progress depends on novel ideas, but current reward systems often fail to recognize them. Many existing metrics conflate novelty with popularity, privileging ideas that fit existing paradigms over those that challenge them. This study develops a theory-driven framework to better understand how different types of novelty emerge, take hold, and receive recognition. Drawing on network scie… ▽ More Scientific progress depends on novel ideas, but current reward systems often fail to recognize them. Many existing metrics conflate novelty with popularity, privileging ideas that fit existing paradigms over those that challenge them. This study develops a theory-driven framework to better understand how different types of novelty emerge, take hold, and receive recognition. Drawing on network science and theories of discovery, we introduce a triadic typology: Pioneers, who introduce entirely new topics; Mavericks, who recombine distant concepts; and Vanguards, who reinforce weak but promising connections. We apply this typology to a dataset of 41,623 articles in the interdisciplinary field of philanthropy and nonprofit studies, linking novelty types to five-year citation counts using mixed-effects negative binomial regression. Results show that novelty is not uniformly rewarded. Pioneer efforts are foundational but often overlooked. Maverick novelty shows consistent citation benefits, particularly rewarded when it displaces prior focus. Vanguard novelty is more likely to gain recognition when it strengthens weakly connected topics, but its citation advantage diminishes as those reinforced nodes become more central. To enable fair comparison across time and domains, we introduce a simulated baseline model. These findings improve the evaluation of innovations, affecting science policy, funding, and institutional assessment practices. △ Less

Submitted 25 June, 2025; v1 submitted 21 June, 2025; originally announced June 2025.

Comments: 27 pages, 3 figures, 5 tables

Flatness After All?

Authors: Neta Shoham, Liron Mor-Yosef, Haim Avron

Abstract: Recent literature has examined the relationship between the curvature of the loss function at minima and generalization, mainly in the context of overparameterized networks. A key observation is that "flat" minima tend to generalize better than "sharp" minima. While this idea is supported by empirical evidence, it has also been shown that deep networks can generalize even with arbitrary sharpness,… ▽ More Recent literature has examined the relationship between the curvature of the loss function at minima and generalization, mainly in the context of overparameterized networks. A key observation is that "flat" minima tend to generalize better than "sharp" minima. While this idea is supported by empirical evidence, it has also been shown that deep networks can generalize even with arbitrary sharpness, as measured by either the trace or the spectral norm of the Hessian. In this paper, we argue that generalization could be assessed by measuring flatness using a soft rank measure of the Hessian. We show that when the common neural network model (neural network with exponential family negative log likelihood loss) is calibrated, and its prediction error and its confidence in the prediction are not correlated with the first and the second derivatives of the network's output, our measure accurately captures the asymptotic expected generalization gap. For non-calibrated models, we connect our flatness measure to the well-known Takeuchi Information Criterion and show that it still provides reliable estimates of generalization gaps for models that are not overly confident. Experimental results indicate that our approach offers a robust estimate of the generalization gap compared to baselines. △ Less

Submitted 21 June, 2025; originally announced June 2025.

A practical identifiability criterion leveraging weak-form parameter estimation

Authors: Nora Heitzman-Breen, Vanja Dukic, David M. Bortz

Abstract: In this work, we define a practical identifiability criterion, (e, q)-identifiability, based on a parameter e, reflecting the noise in observed variables, and a parameter q, reflecting the mean-square error of the parameter estimator. This criterion is better able to encompass changes in the quality of the parameter estimate due to increased noise in the data (compared to existing criteria based s… ▽ More In this work, we define a practical identifiability criterion, (e, q)-identifiability, based on a parameter e, reflecting the noise in observed variables, and a parameter q, reflecting the mean-square error of the parameter estimator. This criterion is better able to encompass changes in the quality of the parameter estimate due to increased noise in the data (compared to existing criteria based solely on average relative errors). Furthermore, we leverage a weak-form equation error-based method of parameter estimation for systems with unobserved variables to assess practical identifiability far more quickly in comparison to output error-based parameter estimation. We do so by generating weak-form input-output equations using differential algebra techniques, as previously proposed by Boulier et al [1], and then applying Weak form Estimation of Nonlinear Dynamics (WENDy) to obtain parameter estimates. This method is computationally efficient and robust to noise, as demonstrated through two classical biological modelling examples. △ Less

Submitted 20 June, 2025; originally announced June 2025.

Differentiable neural network representation of multi-well, locally-convex potentials

Authors: Reese E. Jones, Adrian Buganza Tepole, Jan N. Fuhg

Abstract: Multi-well potentials are ubiquitous in science, modeling phenomena such as phase transitions, dynamic instabilities, and multimodal behavior across physics, chemistry, and biology. In contrast to non-smooth minimum-of-mixture representations, we propose a differentiable and convex formulation based on a log-sum-exponential (LSE) mixture of input convex neural network (ICNN) modes. This log-sum-ex… ▽ More Multi-well potentials are ubiquitous in science, modeling phenomena such as phase transitions, dynamic instabilities, and multimodal behavior across physics, chemistry, and biology. In contrast to non-smooth minimum-of-mixture representations, we propose a differentiable and convex formulation based on a log-sum-exponential (LSE) mixture of input convex neural network (ICNN) modes. This log-sum-exponential input convex neural network (LSE-ICNN) provides a smooth surrogate that retains convexity within basins and allows for gradient-based learning and inference. A key feature of the LSE-ICNN is its ability to automatically discover both the number of modes and the scale of transitions through sparse regression, enabling adaptive and parsimonious modeling. We demonstrate the versatility of the LSE-ICNN across diverse domains, including mechanochemical phase transformations, microstructural elastic instabilities, conservative biological gene circuits, and variational inference for multimodal probability distributions. These examples highlight the effectiveness of the LSE-ICNN in capturing complex multimodal landscapes while preserving differentiability, making it broadly applicable in data-driven modeling, optimization, and physical simulation. △ Less

Submitted 6 June, 2025; originally announced June 2025.

Comments: 16 pages, 13 figures

Optimal Implicit Bias in Linear Regression

Authors: Kanumuri Nithin Varma, Babak Hassibi

Abstract: Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias… ▽ More Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, ``What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter. △ Less

Submitted 20 June, 2025; originally announced June 2025.

Quasi-Monte Carlo with one categorical variable

Authors: Valerie N. P. Ho, Art B. Owen, Zexin Pan

Abstract: We study randomized quasi-Monte Carlo (RQMC) estimation of a multivariate integral where one of the variables takes only a finite number of values. This problem arises when the variable of integration is drawn from a mixture distribution as is common in importance sampling and also arises in some recent work on transport maps. We find that when integration error decreases at an RQMC rate that it i… ▽ More We study randomized quasi-Monte Carlo (RQMC) estimation of a multivariate integral where one of the variables takes only a finite number of values. This problem arises when the variable of integration is drawn from a mixture distribution as is common in importance sampling and also arises in some recent work on transport maps. We find that when integration error decreases at an RQMC rate that it is then beneficial to oversample the smallest mixture components instead of using a proportional allocation. We also find that for the most accurate RQMC sampling methods, it is advantageous to arrange that our $n=2^m$ randomized Sobol' points split into subsample sizes that are also powers of~$2$. △ Less

Submitted 19 June, 2025; originally announced June 2025.

Random feature approximation for general spectral methods

Authors: Mike Nguyen, Nicole Mücke

Abstract: Random feature approximation is arguably one of the most widely used techniques for kernel methods in large-scale learning algorithms. In this work, we analyze the generalization properties of random feature methods, extending previous results for Tikhonov regularization to a broad class of spectral regularization techniques. This includes not only explicit methods but also implicit schemes such a… ▽ More Random feature approximation is arguably one of the most widely used techniques for kernel methods in large-scale learning algorithms. In this work, we analyze the generalization properties of random feature methods, extending previous results for Tikhonov regularization to a broad class of spectral regularization techniques. This includes not only explicit methods but also implicit schemes such as gradient descent and accelerated algorithms like the Heavy-Ball and Nesterov method. Through this framework, we enable a theoretical analysis of neural networks and neural operators through the lens of the Neural Tangent Kernel (NTK) approach trained via gradient descent. For our estimators we obtain optimal learning rates over regularity classes (even for classes that are not included in the reproducing kernel Hilbert space), which are defined through appropriate source conditions. This improves or completes previous results obtained in related settings for specific kernel algorithms. △ Less

Submitted 19 June, 2025; originally announced June 2025.

Comments: arXiv admin note: substantial text overlap with arXiv:2308.15434, arXiv:2412.17518

Convergent Methods for Koopman Operators on Reproducing Kernel Hilbert Spaces

Authors: Nicolas Boullé, Matthew J. Colbrook, Gustav Conradie

Abstract: Data-driven spectral analysis of Koopman operators is a powerful tool for understanding numerous real-world dynamical systems, from neuronal activity to variations in sea surface temperature. The Koopman operator acts on a function space and is most commonly studied on the space of square-integrable functions. However, defining it on a suitable reproducing kernel Hilbert space (RKHS) offers numero… ▽ More Data-driven spectral analysis of Koopman operators is a powerful tool for understanding numerous real-world dynamical systems, from neuronal activity to variations in sea surface temperature. The Koopman operator acts on a function space and is most commonly studied on the space of square-integrable functions. However, defining it on a suitable reproducing kernel Hilbert space (RKHS) offers numerous practical advantages, including pointwise predictions with error bounds, improved spectral properties that facilitate computations, and more efficient algorithms, particularly in high dimensions. We introduce the first general, provably convergent, data-driven algorithms for computing spectral properties of Koopman and Perron--Frobenius operators on RKHSs. These methods efficiently compute spectra and pseudospectra with error control and spectral measures while exploiting the RKHS structure to avoid the large-data limits required in the $L^2$ settings. The function space is determined by a user-specified kernel, eliminating the need for quadrature-based sampling as in $L^2$ and enabling greater flexibility with finite, externally provided datasets. Using the Solvability Complexity Index hierarchy, we construct adversarial dynamical systems for these problems to show that no algorithm can succeed in fewer limits, thereby proving the optimality of our algorithms. Notably, this impossibility extends to randomized algorithms and datasets. We demonstrate the effectiveness of our algorithms on challenging, high-dimensional datasets arising from real-world measurements and high-fidelity numerical simulations, including turbulent channel flow, molecular dynamics of a binding protein, Antarctic sea ice concentration, and Northern Hemisphere sea surface height. The algorithms are publicly available in the software package $\texttt{SpecRKHS}$. △ Less

Submitted 18 June, 2025; originally announced June 2025.

MSC Class: 37A30; 37M10; 37N10; 47A10; 47B32; 47B33; 65P99

Dynamic guessing for Hamiltonian Monte Carlo with embedded numerical root-finding

Authors: Teddy Groves, Nicholas Luke Cowie, Lars Keld Nielsen

Abstract: Modern implementations of Hamiltonian Monte Carlo and related MCMC algorithms support sampling of probability functions that embed numerical root-finding algorithms, thereby allowing fitting of statistical models involving analytically intractable algebraic constraints. However the application of these models in practice is limited by the computational cost of computing large numbers of numerical… ▽ More Modern implementations of Hamiltonian Monte Carlo and related MCMC algorithms support sampling of probability functions that embed numerical root-finding algorithms, thereby allowing fitting of statistical models involving analytically intractable algebraic constraints. However the application of these models in practice is limited by the computational cost of computing large numbers of numerical solutions. We identify a key limitation of previous approaches to HMC with embedded root-finding, which require the starting guess to be the same at all points on the same simulated Hamiltonian trajectory. We demonstrate that this requirement can be relaxed, so that the starting guess depends on the previous integrator state. To choose a good guess using this information we propose two heuristics: use the previous solution and extrapolate the previous solution using implicit differentiation. Both heuristics yield substantial performance improvements on a range of representative models compared with static guessing. We also present grapevine, a JAX-based Python package providing easy access to an implementation of the No-U-Turn sampler augmented with dynamic guessing. △ Less

Submitted 18 June, 2025; originally announced June 2025.

Comments: 10 pages, 2 figures. See https://github.com/dtu-qmcm/grapevine for associated code

Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning

Authors: Wassim Bouaziz, Mathurin Videau, Nicolas Usunier, El-Mahdi El-Mhamdi

Abstract: The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted beha… ▽ More The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) is not only feasible but also allow to effectively protect a dataset and trace its use. Using gradient-based optimization prompt-tuning, we make a model learn arbitrary secret sequences: secret responses to secret prompts that are absent from the training corpus. We validate our approach on language models pre-trained from scratch and show that less than 0.005% of poisoned tokens are sufficient to covertly make a LM learn a secret and detect it with extremely high confidence ($p < 10^{-55}$) with a theoretically certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set. △ Less

Submitted 17 June, 2025; originally announced June 2025.

Comments: 18 pages, 12 figures

On the Hardness of Bandit Learning

Authors: Nataly Brukhim, Aldo Pacchiano, Miroslav Dudik, Robert Schapire

Abstract: We study the task of bandit learning, also known as best-arm identification, under the assumption that the true reward function f belongs to a known, but arbitrary, function class F. We seek a general theory of bandit learnability, akin to the PAC framework for classification. Our investigation is guided by the following two questions: (1) which classes F are learnable, and (2) how they are learna… ▽ More We study the task of bandit learning, also known as best-arm identification, under the assumption that the true reward function f belongs to a known, but arbitrary, function class F. We seek a general theory of bandit learnability, akin to the PAC framework for classification. Our investigation is guided by the following two questions: (1) which classes F are learnable, and (2) how they are learnable. For example, in the case of binary PAC classification, learnability is fully determined by a combinatorial dimension - the VC dimension- and can be attained via a simple algorithmic principle, namely, empirical risk minimization (ERM). In contrast to classical learning-theoretic results, our findings reveal limitations of learning in structured bandits, offering insights into the boundaries of bandit learnability. First, for the question of "which", we show that the paradigm of identifying the learnable classes via a dimension-like quantity fails for bandit learning. We give a simple proof demonstrating that no combinatorial dimension can characterize bandit learnability, even in finite classes, following a standard definition of dimension introduced by Ben-David et al. (2019). For the question of "how", we prove a computational hardness result: we construct a reward function class for which at most two queries are needed to find the optimal action, yet no algorithm can do so in polynomial time unless RP=NP. We also prove that this class admits efficient algorithms for standard algorithmic operations often considered in learning theory, such as an ERM. This implies that computational hardness is in this case inherent to the task of bandit learning. Beyond these results, we investigate additional themes such as learning under noise, trade-offs between noise models, and the relationship between query complexity and regret minimization. △ Less

Submitted 17 June, 2025; originally announced June 2025.

Comments: 13 main pages

Bayesian Hybrid Machine Learning of Gallstone Risk

Authors: Chitradipa Chakraborty, Nayana Mukherjee

Abstract: Gallstone disease is a complex, multifactorial condition with significant global health burdens. Identifying underlying risk factors and their interactions is crucial for early diagnosis, targeted prevention, and effective clinical management. Although logistic regression remains a standard tool for assessing associations between predictors and gallstone status, it often underperforms in high-dime… ▽ More Gallstone disease is a complex, multifactorial condition with significant global health burdens. Identifying underlying risk factors and their interactions is crucial for early diagnosis, targeted prevention, and effective clinical management. Although logistic regression remains a standard tool for assessing associations between predictors and gallstone status, it often underperforms in high-dimensional settings and may fail to capture intricate relationships among variables. To address these limitations, we propose a hybrid machine learning framework that integrates robust variable selection with advanced interaction detection. Specifically, Adaptive LASSO is employed to identify a sparse and interpretable subset of influential features, followed by Bayesian Additive Regression Trees (BART) to model nonlinear effects and uncover key interactions. Selected interactions are further characterized by physiological knowledge through differential equation-informed interaction terms, grounding the model in biologically plausible mechanisms. The insights gained from these steps are then integrated into a final logistic regression model within a Bayesian framework, providing a balance between predictive accuracy and clinical interpretability. This proposed framework not only enhances prediction but also yields actionable insights, offering a valuable support tool for medical research and decision-making. △ Less

Submitted 17 June, 2025; originally announced June 2025.

Comments: 25 pages, 5 figures

hyperFA*IR: A hypergeometric approach to fair rankings with finite candidate pool

Authors: Mauritz N. Cartier van Dissel, Samuel Martin-Gutierrez, Lisette Espín-Noboa, Ana María Jaramillo, Fariba Karimi

Abstract: Ranking algorithms play a pivotal role in decision-making processes across diverse domains, from search engines to job applications. When rankings directly impact individuals, ensuring fairness becomes essential, particularly for groups that are marginalised or misrepresented in the data. Most of the existing group fairness frameworks often rely on ensuring proportional representation of protected… ▽ More Ranking algorithms play a pivotal role in decision-making processes across diverse domains, from search engines to job applications. When rankings directly impact individuals, ensuring fairness becomes essential, particularly for groups that are marginalised or misrepresented in the data. Most of the existing group fairness frameworks often rely on ensuring proportional representation of protected groups. However, these approaches face limitations in accounting for the stochastic nature of ranking processes or the finite size of candidate pools. To this end, we present hyperFA*IR, a framework for assessing and enforcing fairness in rankings drawn from a finite set of candidates. It relies on a generative process based on the hypergeometric distribution, which models real-world scenarios by sampling without replacement from fixed group sizes. This approach improves fairness assessment when top-$k$ selections are large relative to the pool or when protected groups are small. We compare our approach to the widely used binomial model, which treats each draw as independent with fixed probability, and demonstrate$-$both analytically and empirically$-$that our method more accurately reproduces the statistical properties of sampling from a finite population. To operationalise this framework, we propose a Monte Carlo-based algorithm that efficiently detects unfair rankings by avoiding computationally expensive parameter tuning. Finally, we adapt our generative approach to define affirmative action policies by introducing weights into the sampling process. △ Less

Submitted 17 June, 2025; originally announced June 2025.

Comments: In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT'25)

Improving LoRA with Variational Learning

Authors: Bai Cong, Nico Daheim, Yuesong Shen, Rio Yokota, Mohammad Emtiyaz Khan, Thomas Möllenhoff

Abstract: Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can sometimes even be detrimental. Moreover, Bayesian methods also increase computational overheads and require additional tricks for them to work well. Here, we fix these issues by using a recently proposed variational algorit… ▽ More Bayesian methods have recently been used to improve LoRA finetuning and, although they improve calibration, their effect on other metrics (such as accuracy) is marginal and can sometimes even be detrimental. Moreover, Bayesian methods also increase computational overheads and require additional tricks for them to work well. Here, we fix these issues by using a recently proposed variational algorithm called IVON. We show that IVON is easy to implement and has similar costs to AdamW, and yet it can also drastically improve many metrics by using a simple posterior pruning technique. We present extensive results on billion-scale LLMs (Llama and Qwen series) going way beyond the scale of existing applications of IVON. For example, we finetune a Llama-3.2-3B model on a set of commonsense reasoning tasks and improve accuracy over AdamW by 1.3% and reduce ECE by 5.4%, outperforming AdamW and other recent Bayesian methods like Laplace-LoRA and BLoB. Overall, our results show that variational learning with IVON can effectively improve LoRA finetuning. △ Less

Submitted 17 June, 2025; originally announced June 2025.

Comments: 16 pages, 4 figures

SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecasting

Authors: Yitian Zhang, Liheng Ma, Antonios Valkanas, Boris N. Oreshkin, Mark Coates

Abstract: Koopman operator theory provides a framework for nonlinear dynamical system analysis and time-series forecasting by mapping dynamics to a space of real-valued measurement functions, enabling a linear operator representation. Despite the advantage of linearity, the operator is generally infinite-dimensional. Therefore, the objective is to learn measurement functions that yield a tractable finite-di… ▽ More Koopman operator theory provides a framework for nonlinear dynamical system analysis and time-series forecasting by mapping dynamics to a space of real-valued measurement functions, enabling a linear operator representation. Despite the advantage of linearity, the operator is generally infinite-dimensional. Therefore, the objective is to learn measurement functions that yield a tractable finite-dimensional Koopman operator approximation. In this work, we establish a connection between Koopman operator approximation and linear Recurrent Neural Networks (RNNs), which have recently demonstrated remarkable success in sequence modeling. We show that by considering an extended state consisting of lagged observations, we can establish an equivalence between a structured Koopman operator and linear RNN updates. Building on this connection, we present SKOLR, which integrates a learnable spectral decomposition of the input signal with a multilayer perceptron (MLP) as the measurement functions and implements a structured Koopman operator via a highly parallel linear RNN stack. Numerical experiments on various forecasting benchmarks and dynamical systems show that this streamlined, Koopman-theory-based design delivers exceptional performance. △ Less

Submitted 16 June, 2025; originally announced June 2025.

Rademacher learning rates for iterated random functions

Authors: Nikola Sandrić

Abstract: Most existing literature on supervised machine learning assumes that the training dataset is drawn from an i.i.d. sample. However, many real-world problems exhibit temporal dependence and strong correlations between the marginal distributions of the data-generating process, suggesting that the i.i.d. assumption is often unrealistic. In such cases, models naturally include time-series processes wit… ▽ More Most existing literature on supervised machine learning assumes that the training dataset is drawn from an i.i.d. sample. However, many real-world problems exhibit temporal dependence and strong correlations between the marginal distributions of the data-generating process, suggesting that the i.i.d. assumption is often unrealistic. In such cases, models naturally include time-series processes with mixing properties, as well as irreducible and aperiodic ergodic Markov chains. Moreover, the learning rates typically obtained in these settings are independent of the data distribution, which can lead to restrictive choices of hypothesis classes and suboptimal sample complexities for the learning algorithm. In this article, we consider the case where the training dataset is generated by an iterated random function (i.e., an iteratively defined time-homogeneous Markov chain) that is not necessarily irreducible or aperiodic. Under the assumption that the governing function is contractive with respect to its first argument and subject to certain regularity conditions on the hypothesis class, we first establish a uniform convergence result for the corresponding sample error. We then demonstrate the learnability of the approximate empirical risk minimization algorithm and derive its learning rate bound. Both rates are data-distribution dependent, expressed in terms of the Rademacher complexities of the underlying hypothesis class, allowing them to more accurately reflect the properties of the data-generating distribution. △ Less

Submitted 16 June, 2025; originally announced June 2025.

MSC Class: 68W40; 68T10; 60J05

Computational lower bounds in latent models: clustering, sparse-clustering, biclustering

Authors: Bertrand Even, Christophe Giraud, Nicolas Verzelen

Abstract: In many high-dimensional problems, like sparse-PCA, planted clique, or clustering, the best known algorithms with polynomial time complexity fail to reach the statistical performance provably achievable by algorithms free of computational constraints. This observation has given rise to the conjecture of the existence, for some problems, of gaps -- so called statistical-computational gaps -- betwee… ▽ More In many high-dimensional problems, like sparse-PCA, planted clique, or clustering, the best known algorithms with polynomial time complexity fail to reach the statistical performance provably achievable by algorithms free of computational constraints. This observation has given rise to the conjecture of the existence, for some problems, of gaps -- so called statistical-computational gaps -- between the best possible statistical performance achievable without computational constraints, and the best performance achievable with poly-time algorithms. A powerful approach to assess the best performance achievable in poly-time is to investigate the best performance achievable by polynomials with low-degree. We build on the seminal paper of Schramm and Wein (2022) and propose a new scheme to derive lower bounds on the performance of low-degree polynomials in some latent space models. By better leveraging the latent structures, we obtain new and sharper results, with simplified proofs. We then instantiate our scheme to provide computational lower bounds for the problems of clustering, sparse clustering, and biclustering. We also prove matching upper-bounds and some additional statistical results, in order to provide a comprehensive description of the statistical-computational gaps occurring in these three problems. △ Less

Submitted 16 June, 2025; originally announced June 2025.

MSC Class: 62H30; 68Q17

No-Regret Learning Under Adversarial Resource Constraints: A Spending Plan Is All You Need!

Authors: Francesco Emanuele Stradi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti, Christian Kroer

Abstract: We study online decision making problems under resource constraints, where both reward and cost functions are drawn from distributions that may change adversarially over time. We focus on two canonical settings: $(i)$ online resource allocation where rewards and costs are observed before action selection, and $(ii)$ online learning with resource constraints where they are observed after action sel… ▽ More We study online decision making problems under resource constraints, where both reward and cost functions are drawn from distributions that may change adversarially over time. We focus on two canonical settings: $(i)$ online resource allocation where rewards and costs are observed before action selection, and $(ii)$ online learning with resource constraints where they are observed after action selection, under full feedback or bandit feedback. It is well known that achieving sublinear regret in these settings is impossible when reward and cost distributions may change arbitrarily over time. To address this challenge, we analyze a framework in which the learner is guided by a spending plan--a sequence prescribing expected resource usage across rounds. We design general (primal-)dual methods that achieve sublinear regret with respect to baselines that follow the spending plan. Crucially, the performance of our algorithms improves when the spending plan ensures a well-balanced distribution of the budget across rounds. We additionally provide a robust variant of our methods to handle worst-case scenarios where the spending plan is highly imbalanced. To conclude, we study the regret of our algorithms when competing against benchmarks that deviate from the prescribed spending plan. △ Less

Submitted 18 June, 2025; v1 submitted 16 June, 2025; originally announced June 2025.

Finite sample-optimal adjustment sets in linear Gaussian causal models

Authors: Nadja Rutsch, Sara Magliacane, Stéphanie van der Pas

Abstract: Traditional covariate selection methods for causal inference focus on achieving unbiasedness and asymptotic efficiency. In many practical scenarios, researchers must estimate causal effects from observational data with limited sample sizes or in cases where covariates are difficult or costly to measure. Their needs might be better met by selecting adjustment sets that are finite sample-optimal in… ▽ More Traditional covariate selection methods for causal inference focus on achieving unbiasedness and asymptotic efficiency. In many practical scenarios, researchers must estimate causal effects from observational data with limited sample sizes or in cases where covariates are difficult or costly to measure. Their needs might be better met by selecting adjustment sets that are finite sample-optimal in terms of mean squared error. In this paper, we aim to find the adjustment set that minimizes the mean squared error of the causal effect estimator, taking into account the joint distribution of the variables and the sample size. We call this finite sample-optimal set the MSE-optimal adjustment set and present examples in which the MSE-optimal adjustment set differs from the asymptotically optimal adjustment set. To identify the MSE-optimal adjustment set, we then introduce a sample size criterion for comparing adjustment sets in linear Gaussian models. We also develop graphical criteria to reduce the search space for this adjustment set based on the causal graph. In experiments with simulated data, we show that the MSE-optimal adjustment set can outperform the asymptotically optimal adjustment set in finite sample size settings, making causal inference more practical in such scenarios. △ Less

Submitted 15 June, 2025; originally announced June 2025.

Kernel Density Balancing

Authors: John Park, Ning Hao, Yue Selena Niu, Ming Hu

Abstract: High-throughput chromatin conformation capture (Hi-C) data provide insights into the 3D structure of chromosomes, with normalization being a crucial pre-processing step. A common technique for normalization is matrix balancing, which rescales rows and columns of a Hi-C matrix to equalize their sums. Despite its popularity and convenience, matrix balancing lacks statistical justification. In this p… ▽ More High-throughput chromatin conformation capture (Hi-C) data provide insights into the 3D structure of chromosomes, with normalization being a crucial pre-processing step. A common technique for normalization is matrix balancing, which rescales rows and columns of a Hi-C matrix to equalize their sums. Despite its popularity and convenience, matrix balancing lacks statistical justification. In this paper, we introduce a statistical model to analyze matrix balancing methods and propose a kernel-based estimator that leverages spatial structure. Under mild assumptions, we demonstrate that the kernel-based method is consistent, converges faster, and is more robust to data sparsity compared to existing approaches. △ Less

Submitted 14 June, 2025; originally announced June 2025.

Beyond Laplace and Gaussian: Exploring the Generalized Gaussian Mechanism for Private Machine Learning

Authors: Roy Rinberg, Ilia Shumailov, Vikrant Singhal, Rachel Cummings, Nicolas Papernot

Abstract: Differential privacy (DP) is obtained by randomizing a data analysis algorithm, which necessarily introduces a tradeoff between its utility and privacy. Many DP mechanisms are built upon one of two underlying tools: Laplace and Gaussian additive noise mechanisms. We expand the search space of algorithms by investigating the Generalized Gaussian mechanism, which samples the additive noise term $x$… ▽ More Differential privacy (DP) is obtained by randomizing a data analysis algorithm, which necessarily introduces a tradeoff between its utility and privacy. Many DP mechanisms are built upon one of two underlying tools: Laplace and Gaussian additive noise mechanisms. We expand the search space of algorithms by investigating the Generalized Gaussian mechanism, which samples the additive noise term $x$ with probability proportional to $e^{-\frac{| x |}σ^β }$ for some $β\geq 1$. The Laplace and Gaussian mechanisms are special cases of GG for $β=1$ and $β=2$, respectively. In this work, we prove that all members of the GG family satisfy differential privacy, and provide an extension of an existing numerical accountant (the PRV accountant) for these mechanisms. We show that privacy accounting for the GG Mechanism and its variants is dimension independent, which substantially improves computational costs of privacy accounting. We apply the GG mechanism to two canonical tools for private machine learning, PATE and DP-SGD; we show empirically that $β$ has a weak relationship with test-accuracy, and that generally $β=2$ (Gaussian) is nearly optimal. This provides justification for the widespread adoption of the Gaussian mechanism in DP learning, and can be interpreted as a negative result, that optimizing over $β$ does not lead to meaningful improvements in performance. △ Less

Submitted 14 June, 2025; originally announced June 2025.

Scaling Probabilistic Circuits via Monarch Matrices

Authors: Honghua Zhang, Meihua Dang, Benjie Wang, Stefano Ermon, Nanyun Peng, Guy Van den Broeck

Abstract: Probabilistic Circuits (PCs) are tractable representations of probability distributions allowing for exact and efficient computation of likelihoods and marginals. Recent advancements have improved the scalability of PCs either by leveraging their sparse properties or through the use of tensorized operations for better hardware utilization. However, no existing method fully exploits both aspects si… ▽ More Probabilistic Circuits (PCs) are tractable representations of probability distributions allowing for exact and efficient computation of likelihoods and marginals. Recent advancements have improved the scalability of PCs either by leveraging their sparse properties or through the use of tensorized operations for better hardware utilization. However, no existing method fully exploits both aspects simultaneously. In this paper, we propose a novel sparse and structured parameterization for the sum blocks in PCs. By replacing dense matrices with sparse Monarch matrices, we significantly reduce the memory and computation costs, enabling unprecedented scaling of PCs. From a theory perspective, our construction arises naturally from circuit multiplication; from a practical perspective, compared to previous efforts on scaling up tractable probabilistic models, our approach not only achieves state-of-the-art generative modeling performance on challenging benchmarks like Text8, LM1B and ImageNet, but also demonstrates superior scaling behavior, achieving the same performance with substantially less compute as measured by the number of floating-point operations (FLOPs) during training. △ Less

Submitted 14 June, 2025; originally announced June 2025.

Taxonomy of reduction matrices for Graph Coarsening

Authors: Antonin Joly, Nicolas Keriven, Aline Roumy

Abstract: Graph coarsening aims to diminish the size of a graph to lighten its memory footprint, and has numerous applications in graph signal processing and machine learning. It is usually defined using a reduction matrix and a lifting matrix, which, respectively, allows to project a graph signal from the original graph to the coarsened one and back. This results in a loss of information measured by the so… ▽ More Graph coarsening aims to diminish the size of a graph to lighten its memory footprint, and has numerous applications in graph signal processing and machine learning. It is usually defined using a reduction matrix and a lifting matrix, which, respectively, allows to project a graph signal from the original graph to the coarsened one and back. This results in a loss of information measured by the so-called Restricted Spectral Approximation (RSA). Most coarsening frameworks impose a fixed relationship between the reduction and lifting matrices, generally as pseudo-inverses of each other, and seek to define a coarsening that minimizes the RSA. In this paper, we remark that the roles of these two matrices are not entirely symmetric: indeed, putting constraints on the lifting matrix alone ensures the existence of important objects such as the coarsened graph's adjacency matrix or Laplacian. In light of this, in this paper, we introduce a more general notion of reduction matrix, that is not necessarily the pseudo-inverse of the lifting matrix. We establish a taxonomy of ``admissible'' families of reduction matrices, discuss the different properties that they must satisfy and whether they admit a closed-form description or not. We show that, for a fixed coarsening represented by a fixed lifting matrix, the RSA can be further reduced simply by modifying the reduction matrix. We explore different examples, including some based on a constrained optimization process of the RSA. Since this criterion has also been linked to the performance of Graph Neural Networks, we also illustrate the impact of this choices on different node classification tasks on coarsened graphs. △ Less

Submitted 13 June, 2025; originally announced June 2025.

Coefficient Shape Transfer Learning for Functional Linear Regression

Authors: Shuhao Jiao, Ian W. Mckeague, N. -H. Chan

Abstract: In this paper, we develop a novel transfer learning methodology to tackle the challenge of data scarcity in functional linear models. The methodology incorporates samples from the target model (target domain) alongside those from auxiliary models (source domains), transferring knowledge of coefficient shape from the source domains to the target domain. This shape-based knowledge transfer offers tw… ▽ More In this paper, we develop a novel transfer learning methodology to tackle the challenge of data scarcity in functional linear models. The methodology incorporates samples from the target model (target domain) alongside those from auxiliary models (source domains), transferring knowledge of coefficient shape from the source domains to the target domain. This shape-based knowledge transfer offers two key advantages. First, it is robust to covariate scaling, ensuring effectiveness despite variations in data distributions across different source domains. Second, the notion of coefficient shape homogeneity represents a meaningful advance beyond traditional coefficient homogeneity, allowing the method to exploit a wider range of source domains and achieve significantly improved model estimation. We rigorously analyze the convergence rates of the proposed estimator and examine the minimax optimality. Our findings show that the degree of improvement depends not only on the similarity of coefficient shapes between the target and source domains, but also on coefficient magnitudes and the spectral decay rates of the functional covariates covariance operators. To address situations where only a subset of auxiliary models is informative for the target model, we further develop a data-driven procedure for identifying such informative sources. The effectiveness of the proposed methodology is demonstrated through comprehensive simulation studies and an application to occupation time analysis using physical activity data from the U.S. National Health and Nutrition Examination Survey. △ Less

Submitted 12 June, 2025; originally announced June 2025.

Measuring multi-calibration

Authors: Ido Guy, Daniel Haimovich, Fridolin Linder, Nastaran Okati, Lorenzo Perini, Niek Tax, Mark Tygert

Abstract: A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities, the probabilistic predictions are known as "perfectly calibrated." When the predicted probabilities are perfectly calibrated simultaneously across several subpopulations, the probabilistic predictions are known as "perfe… ▽ More A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities, the probabilistic predictions are known as "perfectly calibrated." When the predicted probabilities are perfectly calibrated simultaneously across several subpopulations, the probabilistic predictions are known as "perfectly multi-calibrated." In practice, predicted probabilities are seldom perfectly multi-calibrated, so a statistic measuring the distance from perfect multi-calibration is informative. A recently proposed metric for calibration, based on the classical Kuiper statistic, is a natural basis for a new metric of multi-calibration and avoids well-known problems of metrics based on binning or kernel density estimation. The newly proposed metric weights the contributions of different subpopulations in proportion to their signal-to-noise ratios; data analyses' ablations demonstrate that the metric becomes noisy when omitting the signal-to-noise ratios from the metric. Numerical examples on benchmark data sets illustrate the new metric. △ Less

Submitted 12 June, 2025; originally announced June 2025.

Comments: 25 pages, 12 tables

Nonparametric estimation of an optimal treatment rule with fused randomized trials and missing effect modifiers

Authors: Nicholas Williams, Kara Rudolph, Iván Díaz

Abstract: A fundamental principle of clinical medicine is that a treatment should only be administered to those patients who would benefit from it. Treatment strategies that assign treatment to patients as a function of their individual characteristics are known as dynamic treatment rules. The dynamic treatment rule that optimizes the outcome in the population is called the optimal dynamic treatment rule. R… ▽ More A fundamental principle of clinical medicine is that a treatment should only be administered to those patients who would benefit from it. Treatment strategies that assign treatment to patients as a function of their individual characteristics are known as dynamic treatment rules. The dynamic treatment rule that optimizes the outcome in the population is called the optimal dynamic treatment rule. Randomized clinical trials are considered the gold standard for estimating the marginal causal effect of a treatment on an outcome; they are often not powered to detect heterogeneous treatment effects, and thus, may rarely inform more personalized treatment decisions. The availability of multiple trials studying a common set of treatments presents an opportunity for combining data, often called data-fusion, to better estimate dynamic treatment rules. However, there may be a mismatch in the set of patient covariates measured across trials. We address this problem here; we propose a nonparametric estimator for the optimal dynamic treatment rule that leverages information across the set of randomized trials. We apply the estimator to fused randomized trials of medications for the treatment of opioid use disorder to estimate a treatment rule that would match patient subgroups with the medication that would minimize risk of return to regular opioid use. △ Less

Submitted 12 June, 2025; originally announced June 2025.

Educational Intervention Re-Wires Social Interactions in Isolated Village Networks

Authors: Marios Papamichalis, Laura Forastiere, Edoardo M. Airoldi, Nicholas A. Christakis

Abstract: Social networks shape behavior, disseminate information, and undergird collective action within communities. Consequently, they can be very valuable in the design of effective interventions to improve community well-being. But any exogenous intervention in networked groups, including ones that just involve the provision of information, can also possibly modify the underlying network structure itse… ▽ More Social networks shape behavior, disseminate information, and undergird collective action within communities. Consequently, they can be very valuable in the design of effective interventions to improve community well-being. But any exogenous intervention in networked groups, including ones that just involve the provision of information, can also possibly modify the underlying network structure itself, and some interventions are indeed designed to do so. While social networks obey certain fundamental principles (captured by network-level statistics, such as the degree distribution or transitivity level), they can nevertheless undergo change across time, as people form and break ties with each other within an overall population. Here, using a randomized controlled trial in 110 remote Honduran villages involving 8,331 people, we evaluated the effects of a 22-month public health intervention on pre-existing social network structures. We leverage a two-stage randomized design, where a varying fraction of households received the intervention in each village. In low-dosage villages (5%, 10%, 20%, and 30%) compared to untreated villages (0%), over a two-year period, individuals who received the intervention tended to sever both inbound and outbound ties with untreated individuals whom they previously trusted for health advice. Conversely, in high-dosage villages (50%, 75%, and 100%), treated individuals increased both their inbound and outbound ties. Furthermore, although the intervention was health-focused, it also reshaped broader friendship and financial ties. In aggregate, the imposition of a novel health information regime in rural villages (as a kind of social institution) led to a significant rewiring in individuals particular connections, but it still had a limited effect on the overall global structure of the village-wide social networks. △ Less

Submitted 12 June, 2025; originally announced June 2025.

Comments: MS 18 pages, 3 figures, SI 52 pages, 9 figures, 83 tables

Textual Bayes: Quantifying Uncertainty in LLM-Based Systems

Authors: Brendan Leigh Ross, Noël Vouitsis, Atiyeh Ashari Ghomi, Rasa Hosseinzadeh, Ji Xin, Zhaoyan Liu, Yi Sui, Shiyi Hou, Kin Kwan Leung, Gabriel Loaiza-Ganem, Jesse C. Cresswell

Abstract: Although large language models (LLMs) are becoming increasingly capable of solving challenging real-world tasks, accurately quantifying their uncertainty remains a critical open problem, which limits their applicability in high-stakes domains. This challenge is further compounded by the closed-source, black-box nature of many state-of-the-art LLMs. Moreover, LLM-based systems can be highly sensiti… ▽ More Although large language models (LLMs) are becoming increasingly capable of solving challenging real-world tasks, accurately quantifying their uncertainty remains a critical open problem, which limits their applicability in high-stakes domains. This challenge is further compounded by the closed-source, black-box nature of many state-of-the-art LLMs. Moreover, LLM-based systems can be highly sensitive to the prompts that bind them together, which often require significant manual tuning (i.e., prompt engineering). In this work, we address these challenges by viewing LLM-based systems through a Bayesian lens. We interpret prompts as textual parameters in a statistical model, allowing us to use a small training dataset to perform Bayesian inference over these prompts. This novel perspective enables principled uncertainty quantification over both the model's textual parameters and its downstream predictions, while also incorporating prior beliefs about these parameters expressed in free-form text. To perform Bayesian inference, a difficult problem even for well-studied data modalities, we introduce Metropolis-Hastings through LLM Proposals (MHLP), a novel Markov chain Monte Carlo (MCMC) algorithm that combines prompt optimization techniques with standard MCMC methods. MHLP is a turnkey modification to existing LLM pipelines, including those that rely exclusively on closed-source models. Empirically, we demonstrate that our method yields improvements in both predictive accuracy and uncertainty quantification (UQ) on a range of LLM benchmarks and UQ tasks. More broadly, our work demonstrates a viable path for incorporating methods from the rich Bayesian literature into the era of LLMs, paving the way for more reliable and calibrated LLM-based systems. △ Less

Submitted 11 June, 2025; originally announced June 2025.

Constrained Denoising, Empirical Bayes, and Optimal Transport

Authors: Adam Quinn Jaffe, Nikolaos Ignatiadis, Bodhisattva Sen

Abstract: In the statistical problem of denoising, Bayes and empirical Bayes methods can "overshrink" their output relative to the latent variables of interest. This work is focused on constrained denoising problems which mitigate such phenomena. At the oracle level, i.e., when the latent variable distribution is assumed known, we apply tools from the theory of optimal transport to characterize the solution… ▽ More In the statistical problem of denoising, Bayes and empirical Bayes methods can "overshrink" their output relative to the latent variables of interest. This work is focused on constrained denoising problems which mitigate such phenomena. At the oracle level, i.e., when the latent variable distribution is assumed known, we apply tools from the theory of optimal transport to characterize the solution to (i) variance-constrained, (ii) distribution-constrained, and (iii) general-constrained denoising problems. At the empirical level, i.e., when the latent variable distribution is not known, we use empirical Bayes methodology to estimate these oracle denoisers. Our approach is modular, and transforms any suitable (unconstrained) empirical Bayes denoiser into a constrained empirical Bayes denoiser. We prove explicit rates of convergence for our proposed methodologies, which both extend and sharpen existing asymptotic results that have previously considered only variance constraints. We apply our methodology in two applications: one in astronomy concerning the relative chemical abundances in a large catalog of red-clump stars, and one in baseball concerning minor- and major league batting skill for rookie players. △ Less

Submitted 11 June, 2025; originally announced June 2025.

Comments: 55 pages, 4 figures. Comments welcome

Learning single-index models via harmonic decomposition

Authors: Nirmit Joshi, Hugo Koubbi, Theodor Misiakiewicz, Nathan Srebro

Abstract: We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection $\langle \boldsymbol{w}_*,\boldsymbol{x}\rangle$. Prior work has shown that under Gaussian inputs, the statistical and computational complexity of recovering $\boldsymbol{w}_*$ is governed by the Hermite e… ▽ More We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection $\langle \boldsymbol{w}_*,\boldsymbol{x}\rangle$. Prior work has shown that under Gaussian inputs, the statistical and computational complexity of recovering $\boldsymbol{w}_*$ is governed by the Hermite expansion of the link function. In this paper, we propose a new perspective: we argue that "spherical harmonics" -- rather than "Hermite polynomials" -- provide the natural basis for this problem, as they capture its intrinsic "rotational symmetry". Building on this insight, we characterize the complexity of learning single-index models under arbitrary spherically symmetric input distributions. We introduce two families of estimators -- based on tensor unfolding and online SGD -- that respectively achieve either optimal sample complexity or optimal runtime, and argue that estimators achieving both may not exist in general. When specialized to Gaussian inputs, our theory not only recovers and clarifies existing results but also reveals new phenomena that had previously been overlooked. △ Less

Submitted 11 June, 2025; originally announced June 2025.

Comments: 80 pages

Adversarial Surrogate Risk Bounds for Binary Classification

Authors: Natalie S. Frank

Abstract: A central concern in classification is the vulnerability of machine learning models to adversarial attacks. Adversarial training is one of the most popular techniques for training robust classifiers, which involves minimizing an adversarial surrogate risk. Recent work characterized when a minimizing sequence of an adversarial surrogate risk is also a minimizing sequence of the adversarial classifi… ▽ More A central concern in classification is the vulnerability of machine learning models to adversarial attacks. Adversarial training is one of the most popular techniques for training robust classifiers, which involves minimizing an adversarial surrogate risk. Recent work characterized when a minimizing sequence of an adversarial surrogate risk is also a minimizing sequence of the adversarial classification risk for binary classification -- a property known as adversarial consistency. However, these results do not address the rate at which the adversarial classification risk converges to its optimal value for such a sequence of functions that minimize the adversarial surrogate. This paper provides surrogate risk bounds that quantify that convergence rate. Additionally, we derive distribution-dependent surrogate risk bounds in the standard (non-adversarial) learning setting, that may be of independent interest. △ Less

Submitted 10 June, 2025; originally announced June 2025.

Comments: 37 pages, 2 figures

Know What You Don't Know: Uncertainty Calibration of Process Reward Models

Authors: Young-Jin Park, Kristjan Greenewald, Kaveh Alim, Hao Wang, Navid Azizan

Abstract: Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs). However, we observe that even state-of-the-art PRMs can be poorly calibrated and often overestimate success probabilities. To address this, we present a calibration approach, performed via quantile regression, that adjusts PRM outputs to better align with true success pro… ▽ More Process reward models (PRMs) play a central role in guiding inference-time scaling algorithms for large language models (LLMs). However, we observe that even state-of-the-art PRMs can be poorly calibrated and often overestimate success probabilities. To address this, we present a calibration approach, performed via quantile regression, that adjusts PRM outputs to better align with true success probabilities. Leveraging these calibrated success estimates and their associated confidence bounds, we introduce an \emph{instance-adaptive scaling} (IAS) framework that dynamically adjusts the inference budget based on the estimated likelihood that a partial reasoning trajectory will yield a correct final answer. Unlike conventional methods that allocate a fixed number of reasoning trajectories per query, this approach successfully adapts to each instance and reasoning step when using our calibrated PRMs. Experiments on mathematical reasoning benchmarks show that (i) our PRM calibration method successfully achieves small calibration error, outperforming the baseline methods, (ii) calibration is crucial for enabling effective adaptive scaling, and (iii) the proposed IAS strategy reduces inference costs while maintaining final answer accuracy, utilizing less compute on more confident problems as desired. △ Less

Submitted 10 June, 2025; originally announced June 2025.