-
Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees
Authors:
Edgar Jaber,
Vincent Blot,
Nicolas Brunel,
Vincent Chabridon,
Emmanuel Remy,
Bertrand Iooss,
Didier Lucor,
Mathilde Mougeot,
Alessandro Leite
Abstract:
Gaussian processes (GPs) are a Bayesian machine learning approach widely used to construct surrogate models for the uncertainty quantification of computer simulation codes in industrial applications. It provides both a mean predictor and an estimate of the posterior prediction variance, the latter being used to produce Bayesian credibility intervals. Interpreting these intervals relies on the Gaus…
▽ More
Gaussian processes (GPs) are a Bayesian machine learning approach widely used to construct surrogate models for the uncertainty quantification of computer simulation codes in industrial applications. It provides both a mean predictor and an estimate of the posterior prediction variance, the latter being used to produce Bayesian credibility intervals. Interpreting these intervals relies on the Gaussianity of the simulation model as well as the well-specification of the priors which are not always appropriate. We propose to address this issue with the help of conformal prediction. In the present work, a method for building adaptive cross-conformal prediction intervals is proposed by weighting the non-conformity score with the posterior standard deviation of the GP. The resulting conformal prediction intervals exhibit a level of adaptivity akin to Bayesian credibility sets and display a significant correlation with the surrogate model local approximation error, while being free from the underlying model assumptions and having frequentist coverage guarantees. These estimators can thus be used for evaluating the quality of a GP surrogate model and can assist a decision-maker in the choice of the best prior for the specific application of the GP. The performance of the method is illustrated through a panel of numerical examples based on various reference databases. Moreover, the potential applicability of the method is demonstrated in the context of surrogate modeling of an expensive-to-evaluate simulator of the clogging phenomenon in steam generators of nuclear reactors.
△ Less
Submitted 15 January, 2024;
originally announced January 2024.
-
Adaptive Conformal Prediction by Reweighting Nonconformity Score
Authors:
Salim I. Amoukou,
Nicolas J. B Brunel
Abstract:
Despite attractive theoretical guarantees and practical successes, Predictive Interval (PI) given by Conformal Prediction (CP) may not reflect the uncertainty of a given model. This limitation arises from CP methods using a constant correction for all test points, disregarding their individual uncertainties, to ensure coverage properties. To address this issue, we propose using a Quantile Regressi…
▽ More
Despite attractive theoretical guarantees and practical successes, Predictive Interval (PI) given by Conformal Prediction (CP) may not reflect the uncertainty of a given model. This limitation arises from CP methods using a constant correction for all test points, disregarding their individual uncertainties, to ensure coverage properties. To address this issue, we propose using a Quantile Regression Forest (QRF) to learn the distribution of nonconformity scores and utilizing the QRF's weights to assign more importance to samples with residuals similar to the test point. This approach results in PI lengths that are more aligned with the model's uncertainty. In addition, the weights learnt by the QRF provide a partition of the features space, allowing for more efficient computations and improved adaptiveness of the PI through groupwise conformalization. Our approach enjoys an assumption-free finite sample marginal and training-conditional coverage, and under suitable assumptions, it also ensures conditional coverage. Our methods work for any nonconformity score and are available as a Python package. We conduct experiments on simulated and real-world data that demonstrate significant improvements compared to existing methods.
△ Less
Submitted 31 May, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
-
Local and Regional Counterfactual Rules: Summarized and Robust Recourses
Authors:
Salim I. Amoukou,
Nicolas J. B Brunel
Abstract:
Counterfactual Explanations (CE) face several unresolved challenges, such as ensuring stability, synthesizing multiple CEs, and providing plausibility and sparsity guarantees. From a more practical point of view, recent studies [Pawelczyk et al., 2022] show that the prescribed counterfactual recourses are often not implemented exactly by individuals and demonstrate that most state-of-the-art CE al…
▽ More
Counterfactual Explanations (CE) face several unresolved challenges, such as ensuring stability, synthesizing multiple CEs, and providing plausibility and sparsity guarantees. From a more practical point of view, recent studies [Pawelczyk et al., 2022] show that the prescribed counterfactual recourses are often not implemented exactly by individuals and demonstrate that most state-of-the-art CE algorithms are very likely to fail in this noisy environment. To address these issues, we propose a probabilistic framework that gives a sparse local counterfactual rule for each observation, providing rules that give a range of values capable of changing decisions with high probability. These rules serve as a summary of diverse counterfactual explanations and yield robust recourses. We further aggregate these local rules into a regional counterfactual rule, identifying shared recourses for subgroups of the data. Our local and regional rules are derived from the Random Forest algorithm, which offers statistical guarantees and fidelity to data distribution by selecting recourses in high-density regions. Moreover, our rules are sparse as we first select the smallest set of variables having a high probability of changing the decision. We have conducted experiments to validate the effectiveness of our counterfactual rules in comparison to standard CE and recent similar attempts. Our methods are available as a Python package.
△ Less
Submitted 18 March, 2024; v1 submitted 29 September, 2022;
originally announced September 2022.
-
Robust PCA for Anomaly Detection and Data Imputation in Seasonal Time Series
Authors:
Hong-Lan Botterman,
Julien Roussel,
Thomas Morzadec,
Ali Jabbari,
Nicolas Brunel
Abstract:
We propose a robust principal component analysis (RPCA) framework to recover low-rank and sparse matrices from temporal observations. We develop an online version of the batch temporal algorithm in order to process larger datasets or streaming data. We empirically compare the proposed approaches with different RPCA frameworks and show their effectiveness in practical situations.
We propose a robust principal component analysis (RPCA) framework to recover low-rank and sparse matrices from temporal observations. We develop an online version of the batch temporal algorithm in order to process larger datasets or streaming data. We empirically compare the proposed approaches with different RPCA frameworks and show their effectiveness in practical situations.
△ Less
Submitted 3 August, 2022;
originally announced August 2022.
-
MAPIE: an open-source library for distribution-free uncertainty quantification
Authors:
Vianney Taquet,
Vincent Blot,
Thomas Morzadec,
Louis Lacombe,
Nicolas Brunel
Abstract:
Estimating uncertainties associated with the predictions of Machine Learning (ML) models is of crucial importance to assess their robustness and predictive power. In this submission, we introduce MAPIE (Model Agnostic Prediction Interval Estimator), an open-source Python library that quantifies the uncertainties of ML models for single-output regression and multi-class classification tasks. MAPIE…
▽ More
Estimating uncertainties associated with the predictions of Machine Learning (ML) models is of crucial importance to assess their robustness and predictive power. In this submission, we introduce MAPIE (Model Agnostic Prediction Interval Estimator), an open-source Python library that quantifies the uncertainties of ML models for single-output regression and multi-class classification tasks. MAPIE implements conformal prediction methods, allowing the user to easily compute uncertainties with strong theoretical guarantees on the marginal coverages and with mild assumptions on the model or on the underlying data distribution. MAPIE is hosted on scikit-learn-contrib and is fully "scikit-learn-compatible". As such, it accepts any type of regressor or classifier coming with a scikit-learn API. The library is available at: https://github.com/scikit-learn-contrib/MAPIE/.
△ Less
Submitted 25 July, 2022;
originally announced July 2022.
-
Curvature and Torsion estimation of 3D functional data: A geometric approach to build the mean shape under the Frenet Serret framework
Authors:
Juhyun Park,
Nicolas Brunel,
Perrine Chassat
Abstract:
The analysis of curves has been routinely dealt with using tools from functional data analysis. However its extension to multi-dimensional curves poses a new challenge due to its inherent geometric features that are difficult to capture with the classical approaches that rely on linear approximations. We develop an alternative characterization of a mean that reflects shape variation of the curves.…
▽ More
The analysis of curves has been routinely dealt with using tools from functional data analysis. However its extension to multi-dimensional curves poses a new challenge due to its inherent geometric features that are difficult to capture with the classical approaches that rely on linear approximations. We develop an alternative characterization of a mean that reflects shape variation of the curves. Based on a geometric representation of the curves through the Frenet-Serret ordinary differential equations, we introduce a new definition of mean curvature and mean torsion, as well as mean shape through the notion of mean vector field. This new formulation of the mean for multi-dimensional curves allows us to integrate the parameters for the shape features into the unified functional data modelling framework. We formulate the estimation problem of the functional parameters in a penalized regression and develop an efficient algorithm. We demonstrate our approach with both simulated data and real data examples.
△ Less
Submitted 4 March, 2022;
originally announced March 2022.
-
Consistent Sufficient Explanations and Minimal Local Rules for explaining regression and classification models
Authors:
Salim I. Amoukou,
Nicolas J. B Brunel
Abstract:
To explain the decision of any model, we extend the notion of probabilistic Sufficient Explanations (P-SE). For each instance, this approach selects the minimal subset of features that is sufficient to yield the same prediction with high probability, while removing other features. The crux of P-SE is to compute the conditional probability of maintaining the same prediction. Therefore, we introduce…
▽ More
To explain the decision of any model, we extend the notion of probabilistic Sufficient Explanations (P-SE). For each instance, this approach selects the minimal subset of features that is sufficient to yield the same prediction with high probability, while removing other features. The crux of P-SE is to compute the conditional probability of maintaining the same prediction. Therefore, we introduce an accurate and fast estimator of this probability via random Forests for any data $(\boldsymbol{X}, Y)$ and show its efficiency through a theoretical analysis of its consistency. As a consequence, we extend the P-SE to regression problems. In addition, we deal with non-discrete features, without learning the distribution of $\boldsymbol{X}$ nor having the model for making predictions. Finally, we introduce local rule-based explanations for regression/classification based on the P-SE and compare our approaches w.r.t other explainable AI methods. These methods are available as a Python package at \url{www.github.com/salimamoukou/acv00}.
△ Less
Submitted 14 October, 2022; v1 submitted 8 November, 2021;
originally announced November 2021.
-
Accurate Shapley Values for explaining tree-based models
Authors:
Salim I. Amoukou,
Nicolas J-B. Brunel,
Tangi Salaün
Abstract:
Shapley Values (SV) are widely used in explainable AI, but their estimation and interpretation can be challenging, leading to inaccurate inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we int…
▽ More
Shapley Values (SV) are widely used in explainable AI, but their estimation and interpretation can be challenging, leading to inaccurate inferences and explanations. As a starting point, we remind an invariance principle for SV and derive the correct approach for computing the SV of categorical variables that are particularly sensitive to the encoding used. In the case of tree-based models, we introduce two estimators of Shapley Values that exploit the tree structure efficiently and are more accurate than state-of-the-art methods. Simulations and comparisons are performed with state-of-the-art algorithms and show the practical gain of our approach. Finally, we discuss the limitations of Shapley Values as a local explanation. These methods are available as a Python package.
△ Less
Submitted 31 May, 2023; v1 submitted 7 June, 2021;
originally announced June 2021.
-
Bayesian reconstruction of memories stored in neural networks from their connectivity
Authors:
Sebastian Goldt,
Florent Krzakala,
Lenka Zdeborová,
Nicolas Brunel
Abstract:
The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions. One such question is whether it is possible to reconstruct the information stored in a recurrent network of neurons, given its synaptic connectivity matrix. Here, we address this question by determining when solving such an infer…
▽ More
The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions. One such question is whether it is possible to reconstruct the information stored in a recurrent network of neurons, given its synaptic connectivity matrix. Here, we address this question by determining when solving such an inference problem is theoretically possible in specific attractor network models and by providing a practical algorithm to do so. The algorithm builds on ideas from statistical physics to perform approximate Bayesian inference and is amenable to exact analysis. We study its performance on three different models, compare the algorithm to standard algorithms such as PCA, and explore the limitations of reconstructing stored patterns from synaptic connectivity.
△ Less
Submitted 29 August, 2022; v1 submitted 16 May, 2021;
originally announced May 2021.
-
The Shapley Value of coalition of variables provides better explanations
Authors:
Salim I. Amoukou,
Nicolas J-B. Brunel,
Tangi Salaün
Abstract:
While Shapley Values (SV) are one of the gold standard for interpreting machine learning models, we show that they are still poorly understood, in particular in the presence of categorical variables or of variables of low importance. For instance, we show that the popular practice that consists in summing the SV of dummy variables is false as it provides wrong estimates of all the SV in the model…
▽ More
While Shapley Values (SV) are one of the gold standard for interpreting machine learning models, we show that they are still poorly understood, in particular in the presence of categorical variables or of variables of low importance. For instance, we show that the popular practice that consists in summing the SV of dummy variables is false as it provides wrong estimates of all the SV in the model and implies spurious interpretations. Based on the identification of null and active coalitions, and a coalitional version of the SV, we provide a correct computation and inference of important variables. Moreover, a Python library (All the experiments and simulations can be reproduced with the publicly available library Active Coalition of Variables, https://www.github.com/salimamoukou/acv00) that computes reliably conditional expectations and SV for tree-based models, is implemented and compared with state-of-the-art algorithms on toy models and real data sets.
△ Less
Submitted 6 April, 2022; v1 submitted 24 March, 2021;
originally announced March 2021.
-
Mean curvature and mean shape for multivariate functional data under Frenet-Serret framework
Authors:
Juhyun Park,
Nicolas J-B. Brunel
Abstract:
The analysis of curves has been routinely dealt with using tools from functional data analysis. However its extension to multi-dimensional curves poses a new challenge due to its inherent geometric features that are difficult to capture with the classical approaches that rely on linear approximations. We propose a new framework for functional data as multidimensional curves that allows us to extra…
▽ More
The analysis of curves has been routinely dealt with using tools from functional data analysis. However its extension to multi-dimensional curves poses a new challenge due to its inherent geometric features that are difficult to capture with the classical approaches that rely on linear approximations. We propose a new framework for functional data as multidimensional curves that allows us to extract geometrical features from noisy data. We define a mean through measuring shape variation of the curves. The notion of shape has been used in functional data analysis somewhat intuitively to find a common pattern in one dimensional curves. As a generalization, we directly utilize a geometric representation of the curves through the Frenet-Serret ordinary differential equations and introduce a new definition of mean curvature and mean shape through the mean ordinary differential equation. We formulate the estimation problem in a penalized regression and develop an efficient algorithm. We demonstrate our approach with both simulated data and a real data example.
△ Less
Submitted 26 October, 2019;
originally announced October 2019.
-
Constrained Nonlinear and Mixed Effects Differential Equation Models for Dynamic Cell Polarity Signaling
Authors:
Zhen Xiao,
Nicolas Brunel,
Zhenbiao Yang,
Xinping Cui
Abstract:
The key of tip growth in eukaryotes is the polarized distribution on plasma membrane of a particle named ROP1. This distribution is the result of a positive feedback loop, whose mechanism can be described by a Differential Equation parametrized by two meaningful parameters kpf and knf . We introduce a mechanistic Integro-Differential Equation (IDE) derived from a spatiotemporal model of cell polar…
▽ More
The key of tip growth in eukaryotes is the polarized distribution on plasma membrane of a particle named ROP1. This distribution is the result of a positive feedback loop, whose mechanism can be described by a Differential Equation parametrized by two meaningful parameters kpf and knf . We introduce a mechanistic Integro-Differential Equation (IDE) derived from a spatiotemporal model of cell polarity and we show how this model can be fitted to real data, i.e., ROP1 intensities measured on pollen tubes. At first, we provide an existence and uniqueness result for the solution of our IDE model under certain conditions. Interestingly, this analysis gives a tractable expression for the likelihood, and our approach can be seen as the estimation of a constrained nonlinear model. Moreover, we introduce a population variability by a constrained nonlinear mixed model. We then propose a constrained Least Squares method to fit the model for the single pollen tube case, and two methods, constrained Methods of Moments and constrained Restricted Maximum Likelihood (REML) to fit the model for the multiple pollen tubes case. The performances of all three methods are studied through simulations and are used on an in-house multiple pollen tubes dataset generated at UC Riverside.
△ Less
Submitted 30 April, 2016;
originally announced May 2016.
-
Parametric Estimation of Ordinary Differential Equations with Orthogonality Conditions
Authors:
Nicolas J-B Brunel,
Quentin Clairon,
Florence d'Alche-Buc
Abstract:
Differential equations are commonly used to model dynamical deterministic systems in applications. When statistical parameter estimation is required to calibrate theoretical models to data, classical statistical estimators are often confronted to complex and potentially ill-posed optimization problem. As a consequence, alternative estimators to classical parametric estimators are needed for obtain…
▽ More
Differential equations are commonly used to model dynamical deterministic systems in applications. When statistical parameter estimation is required to calibrate theoretical models to data, classical statistical estimators are often confronted to complex and potentially ill-posed optimization problem. As a consequence, alternative estimators to classical parametric estimators are needed for obtaining reliable estimates. We propose a gradient matching approach for the estimation of parametric Ordinary Differential Equations observed with noise. Starting from a nonparametric proxy of a true solution of the ODE, we build a parametric estimator based on a variational characterization of the solution. As a Generalized Moment Estimator, our estimator must satisfy a set of orthogonal conditions that are solved in the least squares sense. Despite the use of a nonparametric estimator, we prove the root-$n$ consistency and asymptotic normality of the Orthogonal Conditions estimator. We can derive confidence sets thanks to a closed-form expression for the asymptotic variance. Finally, the OC estimator is compared to classical estimators in several (simulated and real) experiments and ODE models in order to show its versatility and relevance with respect to classical Gradient Matching and Nonlinear Least Squares estimators. In particular, we show on a real dataset of influenza infection that the approach gives reliable estimates. Moreover, we show that our approach can deal directly with more elaborated models such as Delay Differential Equation (DDE).
△ Less
Submitted 28 October, 2014;
originally announced October 2014.
-
State and Parameter Estimation of Partially Observed Linear Ordinary Differential Equations with Deterministic Optimal Control
Authors:
Quentin Clairon,
Nicolas Brunel
Abstract:
Ordinary Differential Equations are a simple but powerful framework for modeling complex systems. Parameter estimation from times series can be done by Nonlinear Least Squares (or other classical approaches), but this can give unsatisfactory results because the inverse problem can be ill-posed, even when the differential equation is linear.
Following recent approaches that use approximate soluti…
▽ More
Ordinary Differential Equations are a simple but powerful framework for modeling complex systems. Parameter estimation from times series can be done by Nonlinear Least Squares (or other classical approaches), but this can give unsatisfactory results because the inverse problem can be ill-posed, even when the differential equation is linear.
Following recent approaches that use approximate solutions of the ODE model, we propose a new method that converts parameter estimation into an optimal control problem: our objective is to determine a control and a parameter that are as close as possible to the data. We derive then a criterion that makes a balance between discrepancy with data and with the model, and we minimize it by using optimization in functions spaces: our approach is related to the so-called Deterministic Kalman Filtering, but different from the usual statistical Kalman filtering. e show the root-$n$ consistency and asymptotic normality of the estimators for the parameter and for the states. Experiments in a toy model and in a real case shows that our approach is generally more accurate and more reliable than Nonlinear Least Squares and Generalized Smoothing, even in misspecified cases.
△ Less
Submitted 28 October, 2014;
originally announced October 2014.
-
A Tracking Approach to Parameter Estimation in Linear Ordinary Differential Equations
Authors:
Nicolas Brunel,
Quentin Clairon
Abstract:
Ordinary Differential Equations are widespread tools to model chemical, physical, biological process but they usually rely on parameters which are of critical importance in terms of dynamic and need to be estimated directly from the data. Classical statistical approaches (nonlinear least squares, maximum likelihood estimator) can give unsatisfactory results because of computational difficulties an…
▽ More
Ordinary Differential Equations are widespread tools to model chemical, physical, biological process but they usually rely on parameters which are of critical importance in terms of dynamic and need to be estimated directly from the data. Classical statistical approaches (nonlinear least squares, maximum likelihood estimator) can give unsatisfactory results because of computational difficulties and ill-posedness of the statistical problem. New estimation methods that use some nonparametric devices have been proposed to circumvent these issues. We present a new estimator that shares properties with Two-Step estimator and Generalized Smoothing (introduced by Ramsay et al, 2007). We introduce a perturbed model and we use optimal control theory for constructing a criterion that aims at minimizing the discrepancy with data and the model. Here, we focus on the case of linear Ordinary Differential Equations as our criterion has a closed-form expression that permits a detailed analysis. Our approach avoids the use of a nonparametric estimator of the derivative, which is one of the main cause of inaccuracy in Two-Step estimators. Moreover, we take into account model discrepancy and our estimator is more robust to model misspecification than classical methods. The discrepancy with the parametric ODE model correspond to the minimum perturbation (or control) to apply to the initial model. Its qualitative analysis can be informative for misspecification diagnosis. In the case of well-specified model, we show the consistency of our estimator and that we reach the parametric root-n rate when regression splines are used in the first step.
△ Less
Submitted 28 October, 2014;
originally announced October 2014.
