-
The Bayesian Infinitesimal Jackknife for Variance
Authors:
Ryan Giordano,
Tamara Broderick
Abstract:
The frequentist variability of Bayesian posterior expectations can provide meaningful measures of uncertainty even when models are misspecified. Classical methods to asymptotically approximate the frequentist covariance of Bayesian estimators such as the Laplace approximation and the nonparametric bootstrap can be practically inconvenient, since the Laplace approximation may require an intractable…
▽ More
The frequentist variability of Bayesian posterior expectations can provide meaningful measures of uncertainty even when models are misspecified. Classical methods to asymptotically approximate the frequentist covariance of Bayesian estimators such as the Laplace approximation and the nonparametric bootstrap can be practically inconvenient, since the Laplace approximation may require an intractable integral to compute the marginal log posterior, and the bootstrap requires computing the posterior for many different bootstrap datasets. We develop and explore the infinitesimal jackknife (IJ), an alternative method for computing asymptotic frequentist covariance of smooth functionals of exchangeable data, which is based on the "influence function" of robust statistics. We show that the influence function for posterior expectations has the form of a simple posterior covariance, and that the IJ covariance estimate is, in turn, easily computed from a single set of posterior samples. Under conditions similar to those required for a Bayesian central limit theorem to apply, we prove that the corresponding IJ covariance estimate is asymptotically equivalent to the Laplace approximation and the bootstrap. In the presence of nuisance parameters that may not obey a central limit theorem, we argue using a von Mises expansion that the IJ covariance is inconsistent, but can remain a good approximation to the limiting frequentist variance. We demonstrate the accuracy and computational benefits of the IJ covariance estimates with simulated and real-world experiments.
△ Less
Submitted 26 June, 2024; v1 submitted 10 May, 2023;
originally announced May 2023.
-
How good is your Laplace approximation of the Bayesian posterior? Finite-sample computable error bounds for a variety of useful divergences
Authors:
Mikołaj J. Kasprzak,
Ryan Giordano,
Tamara Broderick
Abstract:
The Laplace approximation is a popular method for constructing a Gaussian approximation to the Bayesian posterior and thereby approximating the posterior mean and variance. But approximation quality is a concern. One might consider using rate-of-convergence bounds from certain versions of the Bayesian Central Limit Theorem (BCLT) to provide quality guarantees. But existing bounds require assumptio…
▽ More
The Laplace approximation is a popular method for constructing a Gaussian approximation to the Bayesian posterior and thereby approximating the posterior mean and variance. But approximation quality is a concern. One might consider using rate-of-convergence bounds from certain versions of the Bayesian Central Limit Theorem (BCLT) to provide quality guarantees. But existing bounds require assumptions that are unrealistic even for relatively simple real-life Bayesian analyses; more specifically, existing bounds either (1) require knowing the true data-generating parameter, (2) are asymptotic in the number of samples, (3) do not control the Bayesian posterior mean, or (4) require strongly log concave models to compute. In this work, we provide the first computable bounds on quality that simultaneously (1) do not require knowing the true parameter, (2) apply to finite samples, (3) control posterior means and variances, and (4) apply generally to models that satisfy the conditions of the asymptotic BCLT. Moreover, we substantially improve the dimension dependence of existing bounds; in fact, we achieve the lowest-order dimension dependence possible in the general case. We compute exact constants in our bounds for a variety of standard models, including logistic regression, and numerically demonstrate their utility. We provide a framework for analysis of more complex models.
△ Less
Submitted 15 June, 2025; v1 submitted 29 September, 2022;
originally announced September 2022.
-
A Higher-Order Swiss Army Infinitesimal Jackknife
Authors:
Ryan Giordano,
Michael I. Jordan,
Tamara Broderick
Abstract:
Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-consuming. For sufficiently regular optimization problems the optimum depends smooth…
▽ More
Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-consuming. For sufficiently regular optimization problems the optimum depends smoothly on the data weights, and so the process of repeatedly re-fitting can be approximated with a Taylor series that can be often evaluated relatively quickly. The first-order approximation is known as the "infinitesimal jackknife" in the statistics literature and has been the subject of recent interest in machine learning for approximate CV. In this work, we consider high-order approximations, which we call the "higher-order infinitesimal jackknife" (HOIJ). Under mild regularity conditions, we provide a simple recursive procedure to compute approximations of all orders with finite-sample accuracy bounds. Additionally, we show that the HOIJ can be efficiently computed even in high dimensions using forward-mode automatic differentiation. We show that a linear approximation with bootstrap weights approximation is equivalent to those provided by asymptotic normal approximations. Consequently, the HOIJ opens up the possibility of enjoying higher-order accuracy properties of the bootstrap using local approximations. Consistency of the HOIJ for leave-one-out CV under different asymptotic regimes follows as corollaries from our finite-sample bounds under additional regularity assumptions. The generality of the computation and bounds motivate the name "higher-order Swiss Army infinitesimal jackknife."
△ Less
Submitted 28 July, 2019;
originally announced July 2019.
-
Rigidity Theory in SE(2) for Unscaled Relative Position Estimation using only Bearing Measurements
Authors:
Daniel Zelazo,
Antonio Franchi,
Paolo Robuffo Giordano
Abstract:
This work considers the problem of estimating the unscaled relative positions of a multi-robot team in a common reference frame from bearing-only measurements. Each robot has access to a relative bearing measurement taken from the local body frame of the robot, and the robots have no knowledge of a common or inertial reference frame. A corresponding extension of rigidity theory is made for framewo…
▽ More
This work considers the problem of estimating the unscaled relative positions of a multi-robot team in a common reference frame from bearing-only measurements. Each robot has access to a relative bearing measurement taken from the local body frame of the robot, and the robots have no knowledge of a common or inertial reference frame. A corresponding extension of rigidity theory is made for frameworks embedded in the \emph{special Euclidean group} $SE(2) = \mathbb{R}^2 \times \mathcal{S}^1$. We introduce definitions describing rigidity for $SE(2)$ frameworks and provide necessary and sufficient conditions for when such a framework is \emph{infinitesimally rigid} in $SE(2)$. Analogous to the rigidity matrix for point formations, we introduce the \emph{directed bearing rigidity matrix} and show that an $SE(2)$ framework is infinitesimally rigid if and only if the rank of this matrix is equal to $2|\mathcal{V}|-4$, where $|\mathcal{V}|$ is the number of agents in the ensemble. The directed bearing rigidity matrix and its properties are then used in the implementation and convergence proof of a distributed estimator to determine the {unscaled}{} relative positions in a common frame. Some simulation results are also given to support the analysis.
△ Less
Submitted 5 November, 2013;
originally announced November 2013.
-
Decentralized Rigidity Maintenance Control with Range Measurements for Multi-Robot Systems
Authors:
Daniel Zelazo,
Antonio Franchi,
Heinrich H. Bülthoff,
Paolo Robuffo Giordano
Abstract:
This work proposes a fully decentralized strategy for maintaining the formation rigidity of a multi-robot system using only range measurements, while still allowing the graph topology to change freely over time. In this direction, a first contribution of this work is an extension of rigidity theory to weighted frameworks and the rigidity eigenvalue, which when positive ensures the infinitesimal ri…
▽ More
This work proposes a fully decentralized strategy for maintaining the formation rigidity of a multi-robot system using only range measurements, while still allowing the graph topology to change freely over time. In this direction, a first contribution of this work is an extension of rigidity theory to weighted frameworks and the rigidity eigenvalue, which when positive ensures the infinitesimal rigidity of the framework. We then propose a distributed algorithm for estimating a common relative position reference frame amongst a team of robots with only range measurements in addition to one agent endowed with the capability of measuring the bearing to two other agents. This first estimation step is embedded into a subsequent distributed algorithm for estimating the rigidity eigenvalue associated with the weighted framework. The estimate of the rigidity eigenvalue is finally used to generate a local control action for each agent that both maintains the rigidity property and enforces additional con- straints such as collision avoidance and sensing/communication range limits and occlusions. As an additional feature of our approach, the communication and sensing links among the robots are also left free to change over time while preserving rigidity of the whole framework. The proposed scheme is then experimentally validated with a robotic testbed consisting of 6 quadrotor UAVs operating in a cluttered environment.
△ Less
Submitted 4 September, 2014; v1 submitted 2 September, 2013;
originally announced September 2013.
-
Online Leader Selection for Improved Collective Tracking and Formation Maintenance
Authors:
Antonio Franchi,
Paolo Robuffo Giordano
Abstract:
The goal of this work is to propose an extension of the popular leader-follower framework for multi-agent collective tracking and formation maintenance in presence of a time- varying leader. In particular, the leader is persistently selected online so as to optimize the tracking performance of an exogenous collective velocity command while also maintaining a desired formation via a (possibly time-…
▽ More
The goal of this work is to propose an extension of the popular leader-follower framework for multi-agent collective tracking and formation maintenance in presence of a time- varying leader. In particular, the leader is persistently selected online so as to optimize the tracking performance of an exogenous collective velocity command while also maintaining a desired formation via a (possibly time-varying) communication-graph topology. The effects of a change in the leader identity are theoretically analyzed and exploited for defining a suitable error metric able to capture the tracking performance of the multi- agent group. Both the group performance and the metric design are found to depend upon the spectral properties of a special directed graph induced by the identity of the chosen leader. By exploiting these results, as well as distributed estimation techniques, we are then able to detail a fully-decentralized adaptive strategy able to periodically select online the best leader among the neighbors of the current leader. Numerical simulations show that the application of the proposed technique results in an improvement of the overall performance of the group behavior w.r.t. other possible strategies.
△ Less
Submitted 31 January, 2016; v1 submitted 24 May, 2013;
originally announced May 2013.
