-
Practical Equivariances via Relational Conditional Neural Processes
Authors:
Daolang Huang,
Manuel Haussmann,
Ulpu Remes,
ST John,
Grégoire Clarté,
Kevin Sebastian Luck,
Samuel Kaski,
Luigi Acerbi
Abstract:
Conditional Neural Processes (CNPs) are a class of metalearning models popular for combining the runtime efficiency of amortized inference with reliable uncertainty quantification. Many relevant machine learning tasks, such as in spatio-temporal modeling, Bayesian Optimization and continuous control, inherently contain equivariances -- for example to translation -- which the model can exploit for…
▽ More
Conditional Neural Processes (CNPs) are a class of metalearning models popular for combining the runtime efficiency of amortized inference with reliable uncertainty quantification. Many relevant machine learning tasks, such as in spatio-temporal modeling, Bayesian Optimization and continuous control, inherently contain equivariances -- for example to translation -- which the model can exploit for maximal performance. However, prior attempts to include equivariances in CNPs do not scale effectively beyond two input dimensions. In this work, we propose Relational Conditional Neural Processes (RCNPs), an effective approach to incorporate equivariances into any neural process model. Our proposed method extends the applicability and impact of equivariant neural processes to higher dimensions. We empirically demonstrate the competitive performance of RCNPs on a large array of tasks naturally containing equivariances.
△ Less
Submitted 5 November, 2023; v1 submitted 19 June, 2023;
originally announced June 2023.
-
Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature
Authors:
Chengkun Li,
Grégoire Clarté,
Martin Jørgensen,
Luigi Acerbi
Abstract:
In applied Bayesian inference scenarios, users may have access to a large number of pre-existing model evaluations, for example from maximum-a-posteriori (MAP) optimization runs. However, traditional approximate inference techniques make little to no use of this available information. We propose the framework of post-process Bayesian inference as a means to obtain a quick posterior approximation f…
▽ More
In applied Bayesian inference scenarios, users may have access to a large number of pre-existing model evaluations, for example from maximum-a-posteriori (MAP) optimization runs. However, traditional approximate inference techniques make little to no use of this available information. We propose the framework of post-process Bayesian inference as a means to obtain a quick posterior approximation from existing target density evaluations, with no further model calls. Within this framework, we introduce Variational Sparse Bayesian Quadrature (VSBQ), a method for post-process approximate inference for models with black-box and potentially noisy likelihoods. VSBQ reuses existing target density evaluations to build a sparse Gaussian process (GP) surrogate model of the log posterior density function. Subsequently, we leverage sparse-GP Bayesian quadrature combined with variational inference to achieve fast approximate posterior inference over the surrogate. We validate our method on challenging synthetic scenarios and real-world applications from computational neuroscience. The experiments show that VSBQ builds high-quality posterior approximations by post-processing existing optimization traces, with no further model evaluations.
△ Less
Submitted 29 November, 2024; v1 submitted 9 March, 2023;
originally announced March 2023.
-
Online simulator-based experimental design for cognitive model selection
Authors:
Alexander Aushev,
Aini Putkonen,
Gregoire Clarte,
Suyog Chandramouli,
Luigi Acerbi,
Samuel Kaski,
Andrew Howes
Abstract:
The problem of model selection with a limited number of experimental trials has received considerable attention in cognitive science, where the role of experiments is to discriminate between theories expressed as computational models. Research on this subject has mostly been restricted to optimal experiment design with analytically tractable models. However, cognitive models of increasing complexi…
▽ More
The problem of model selection with a limited number of experimental trials has received considerable attention in cognitive science, where the role of experiments is to discriminate between theories expressed as computational models. Research on this subject has mostly been restricted to optimal experiment design with analytically tractable models. However, cognitive models of increasing complexity, with intractable likelihoods, are becoming more commonplace. In this paper, we propose BOSMOS: an approach to experimental design that can select between computational models without tractable likelihoods. It does so in a data-efficient manner, by sequentially and adaptively generating informative experiments. In contrast to previous approaches, we introduce a novel simulator-based utility objective for design selection, and a new approximation of the model likelihood for model selection. In simulated experiments, we demonstrate that the proposed BOSMOS technique can accurately select models in up to 2 orders of magnitude less time than existing LFI alternatives for three cognitive science tasks: memory retention, sequential signal detection and risky choice.
△ Less
Submitted 3 March, 2023;
originally announced March 2023.
-
A Phylogenetic Model of the Evolution of Discrete Matrices for the Joint Inference of Lexical and Phonological Language Histories
Authors:
Grégoire Clarté,
Robin J. Ryder
Abstract:
We propose a model of the evolution of a matrix along a phylogenetic tree, in which transformations affect either entire rows or columns of the matrix. This represents the change of both lexical and phonological aspects of linguistic data, by allowing for new words to appear and for systematic phonological changes to affect the entire vocabulary. We implement a Sequential Monte Carlo method to sam…
▽ More
We propose a model of the evolution of a matrix along a phylogenetic tree, in which transformations affect either entire rows or columns of the matrix. This represents the change of both lexical and phonological aspects of linguistic data, by allowing for new words to appear and for systematic phonological changes to affect the entire vocabulary. We implement a Sequential Monte Carlo method to sample from the posterior distribution, and infer jointly the phylogeny, model parameters, and latent variables representing cognate births and phonological transformations. We successfully apply this method to synthetic and real data of moderate size.
△ Less
Submitted 24 June, 2022;
originally announced June 2022.
-
Lagged couplings diagnose Markov chain Monte Carlo phylogenetic inference
Authors:
Luke J. Kelly,
Robin J. Ryder,
Grégoire Clarté
Abstract:
Phylogenetic inference is an intractable statistical problem on a complex space. Markov chain Monte Carlo methods are the primary tool for Bayesian phylogenetic inference but it is challenging to construct efficient schemes to explore the associated posterior distribution or assess their performance. Existing approaches are unable to diagnose mixing or convergence of Markov schemes jointly across…
▽ More
Phylogenetic inference is an intractable statistical problem on a complex space. Markov chain Monte Carlo methods are the primary tool for Bayesian phylogenetic inference but it is challenging to construct efficient schemes to explore the associated posterior distribution or assess their performance. Existing approaches are unable to diagnose mixing or convergence of Markov schemes jointly across all components of a phylogenetic model. Lagged couplings of Markov chain Monte Carlo algorithms have recently been developed on simpler spaces to diagnose convergence and construct unbiased estimators. We describe a contractive coupling of Markov chains targeting a posterior distribution over a space of phylogenetic trees with branch lengths, scalar parameters and latent variables. We use these couplings to assess mixing and convergence of Markov chains jointly across all components of the phylogenetic model on trees with up to 200 leaves. Samples from our coupled chains may also be used to construct unbiased estimators.
△ Less
Submitted 8 October, 2022; v1 submitted 30 August, 2021;
originally announced August 2021.
-
Collective Proposal Distributions for Nonlinear MCMC samplers: Mean-Field Theory and Fast Implementation
Authors:
Grégoire Clarté,
Antoine Diez,
Jean Feydy
Abstract:
Over the last decades, various "non-linear" MCMC methods have arisen. While appealing for their convergence speed and efficiency, their practical implementation and theoretical study remain challenging. In this paper, we introduce a non-linear generalization of the Metropolis-Hastings algorithm to a proposal that depends not only on the current state, but also on its law. We propose to simulate th…
▽ More
Over the last decades, various "non-linear" MCMC methods have arisen. While appealing for their convergence speed and efficiency, their practical implementation and theoretical study remain challenging. In this paper, we introduce a non-linear generalization of the Metropolis-Hastings algorithm to a proposal that depends not only on the current state, but also on its law. We propose to simulate this dynamics as the mean field limit of a system of interacting particles, that can in turn itself be understood as a generalisation of the Metropolis-Hastings algorithm to a population of particles. Under the double limit in number of iterations and number of particles we prove that this algorithm converges. Then, we propose an efficient GPU implementation and illustrate its performance on various examples. The method is particularly stable on multimodal examples and converges faster than the classical methods.
△ Less
Submitted 2 August, 2022; v1 submitted 18 September, 2019;
originally announced September 2019.
-
Component-wise approximate Bayesian computation via Gibbs-like steps
Authors:
Grégoire Clarté,
Christian P. Robert,
Robin Ryder,
Julien Stoehr
Abstract:
Approximate Bayesian computation methods are useful for generative models with intractable likelihoods. These methods are however sensitive to the dimension of the parameter space, requiring exponentially increasing resources as this dimension grows. To tackle this difficulty, we explore a Gibbs version of the ABC approach that runs component-wise approximate Bayesian computation steps aimed at th…
▽ More
Approximate Bayesian computation methods are useful for generative models with intractable likelihoods. These methods are however sensitive to the dimension of the parameter space, requiring exponentially increasing resources as this dimension grows. To tackle this difficulty, we explore a Gibbs version of the ABC approach that runs component-wise approximate Bayesian computation steps aimed at the corresponding conditional posterior distributions, and based on summary statistics of reduced dimensions. While lacking the standard justifications for the Gibbs sampler, the resulting Markov chain is shown to converge in distribution under some partial independence conditions. The associated stationary distribution can further be shown to be close to the true posterior distribution and some hierarchical versions of the proposed mechanism enjoy a closed form limiting distribution. Experiments also demonstrate the gain in efficiency brought by the Gibbs version over the standard solution.
△ Less
Submitted 17 September, 2020; v1 submitted 31 May, 2019;
originally announced May 2019.
