Showing 1–50 of 55 results for author: Kappen, H

Fröhlich versus Bose-Einstein Condensation in Pumped Bosonic Systems

Authors: Wenhao Xu, Andrey A. Bagrov, Farhan T. Chowdhury, Luke D. Smith, Daniel R. Kattnig, Hilbert J. Kappen, Mikhail I. Katsnelson

Abstract: Magnon-condensation, which emerges in pumped bosonic systems at room temperature, continues to garner great interest for its long-lived coherence. While traditionally formulated in terms of Bose-Einstein condensation, which typically occurs at ultra-low temperatures, it could potentially also be explained by Fröhlich-condensation, a hypothesis of Bose-Einstein-like condensation in living systems a… ▽ More Magnon-condensation, which emerges in pumped bosonic systems at room temperature, continues to garner great interest for its long-lived coherence. While traditionally formulated in terms of Bose-Einstein condensation, which typically occurs at ultra-low temperatures, it could potentially also be explained by Fröhlich-condensation, a hypothesis of Bose-Einstein-like condensation in living systems at ambient temperatures. This prompts general questions relating to fundamental differences between coherence phenomena in open and isolated quantum systems. To that end, we introduce a simple model of bosonic condensation in an open quantum system (OQS) formulation, wherein bosons dissipatively interact with an oscillator (phonon) bath. Our derived equations of motion for expected boson occupations turns out to be similar in form to the rate equations governing Fröhlich-condensation. Provided that specific system parameters result in correlations that amplify or diminish the condensation effects, we thereby posit that our treatment offers a better description of high-temperature condensation compared to traditional formulations obtained using equilibrium thermodynamics. By comparing our OQS derivation with the original uncorrelated and previous semi-classical rate equations, we furthermore highlight how both classical anti-correlations and quantum correlations alter the bosonic occupation distribution. △ Less

Submitted 16 January, 2025; v1 submitted 30 October, 2024; originally announced November 2024.

Comments: 7 pages, 2 figures, plus supplementary material (8 pages)

Journal ref: Phys. Rev. Research 7, 023111 (2025)

Stochastic optimal control of open quantum systems

Authors: Aarón Villanueva, Hilbert Kappen

Abstract: We address the generic problem of optimal quantum state preparation for open quantum systems. It is well known that open quantum systems can be simulated by quantum trajectories described by a stochastic Schrödinger equation. In this context, the state preparation becomes a stochastic optimal control (SOC) problem. The latter requires the solution of the Hamilton-Jacobi-Bellman equation, which is,… ▽ More We address the generic problem of optimal quantum state preparation for open quantum systems. It is well known that open quantum systems can be simulated by quantum trajectories described by a stochastic Schrödinger equation. In this context, the state preparation becomes a stochastic optimal control (SOC) problem. The latter requires the solution of the Hamilton-Jacobi-Bellman equation, which is, in general, challenging to solve. A notable exception are the so-called path integral (PI) control problems, for which one can estimate the optimal control solution by direct sampling of the cost objective. In this work, we derive a class of quantum state preparation problems that are amenable to PI control techniques, and propose a corresponding algorithm, which we call Quantum Diffusion Control (QDC). Unlike conventional quantum control algorithms, QDC avoids computing gradients of the cost function to determine the optimal control. Instead, it employs adaptive importance sampling, a technique where the controls are iteratively improved based on global averages over quantum trajectories. We also demonstrate that QDC, used as an annealer in the environmental coupling strength, finds high accuracy solutions for unitary (noiseless) quantum control problems. We further discuss the implementation of this technique on quantum hardware. We illustrate the effectiveness of our approach through examples of open-loop control for single- and multi-qubit systems. △ Less

Submitted 30 January, 2025; v1 submitted 24 October, 2024; originally announced October 2024.

Comments: We added a new section on annealing control, supplementary material, and formatted for journal submission

Stochastic syncing in sinusoidally driven atomic orbital memory

Authors: Werner M. J. van Weerdenburg, Hermann Osterhage, Ruben Christianen, Kira Junghans, Eduardo Domínguez, Hilbert J. Kappen, Alexander Ako Khajetoorians

Abstract: Stochastically fluctuating multi-well systems as physical implementations of energy-based machine learning models promise a route towards neuromorphic hardware. Understanding the response of multi-well systems to dynamic input signals is crucial in this regard. Here, we investigate the stochastic response of binary orbital memory states derived from individual Fe and Co atoms on a black phosphorus… ▽ More Stochastically fluctuating multi-well systems as physical implementations of energy-based machine learning models promise a route towards neuromorphic hardware. Understanding the response of multi-well systems to dynamic input signals is crucial in this regard. Here, we investigate the stochastic response of binary orbital memory states derived from individual Fe and Co atoms on a black phosphorus surface to sinusoidal input voltages. Using scanning tunneling microscopy, we quantify the state residence times for DC and AC voltage drive with various input frequencies. We find that Fe and Co atoms both exhibit features of synchronization to the AC input, but only Fe atoms demonstrate a significant frequency-dependent change in the time-averaged state occupations. By modeling the underlying stochastic process, we show that the frequency response of the system is directly related to the DC voltage dependence of the state asymmetry. This relation provides a tunable way to induce population changes in stochastic systems and lays the foundation for understanding the response of multi-well systems to dynamical input signals. △ Less

Submitted 29 September, 2023; originally announced September 2023.

Training Quantum Boltzmann Machines with the $β$-Variational Quantum Eigensolver

Authors: Onno Huijgen, Luuk Coopmans, Peyman Najafi, Marcello Benedetti, Hilbert J. Kappen

Abstract: The quantum Boltzmann machine (QBM) is a generative machine learning model for both classical data and quantum states. Training the QBM consists of minimizing the relative entropy from the model to the target state. This requires QBM expectation values which are computationally intractable for large models in general. It is therefore important to develop heuristic training methods that work well i… ▽ More The quantum Boltzmann machine (QBM) is a generative machine learning model for both classical data and quantum states. Training the QBM consists of minimizing the relative entropy from the model to the target state. This requires QBM expectation values which are computationally intractable for large models in general. It is therefore important to develop heuristic training methods that work well in practice. In this work, we study a heuristic method characterized by a nested loop: the inner loop trains the $β$-variational quantum eigensolver ($β$-VQE) by Liu et al (2021 Mach. Learn.: Sci. Technol.2 025011) to approximate the QBM expectation values; the outer loop trains the QBM to minimize the relative entropy to the target. We show that low-rank representations obtained by $β$-VQE provide an efficient way to learn low-rank target states, such as classical data and low-temperature quantum tomography. We test the method on both classical and quantum target data with numerical simulations of up to 10 qubits. For the cases considered here, the obtained QBMs can model the target to high fidelity. We implement a trained model on a physical quantum device. The approach offers a valuable route towards variationally training QBMs on near-term quantum devices. △ Less

Submitted 23 May, 2024; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: 9 pages, 11 figures

Journal ref: Mach. Learn.: Sci. Technol. 5 (2024) 025017

Why adiabatic quantum annealing is unlikely to yield speed-up

Authors: Aarón Villanueva, Peyman Najafi, Hilbert J. Kappen

Abstract: We study quantum annealing for combinatorial optimization with Hamiltonian $H = z H_f + H_0$ where $H_f$ is diagonal, $H_0=-|φ\rangle \langle φ|$ is the equal superposition state projector and $z$ the annealing parameter. We analytically compute the minimal spectral gap as $\mathcal{O}(1/\sqrt{N})$ with $N$ the total number of states and its location $z_*$. We show that quantum speed-up requires a… ▽ More We study quantum annealing for combinatorial optimization with Hamiltonian $H = z H_f + H_0$ where $H_f$ is diagonal, $H_0=-|φ\rangle \langle φ|$ is the equal superposition state projector and $z$ the annealing parameter. We analytically compute the minimal spectral gap as $\mathcal{O}(1/\sqrt{N})$ with $N$ the total number of states and its location $z_*$. We show that quantum speed-up requires an annealing schedule which demands a precise knowledge of $z_*$, which can be computed only if the density of states of the optimization problem is known. However, in general the density of states is intractable to compute, making quadratic speed-up unfeasible for any practical combinatoric optimization problems. We conjecture that it is likely that this negative result also applies for any other instance independent transverse Hamiltonians such as $H_0 = -\sum_{i=1}^n σ_i^x$. △ Less

Submitted 23 October, 2023; v1 submitted 27 December, 2022; originally announced December 2022.

Comments: 23 pages, 6 figures, updated to published version

Efficient inference in the transverse field Ising model

Authors: E. Domínguez, H. J. Kappen

Abstract: In this paper we introduce an approximate method to solve the quantum cavity equations for transverse field Ising models. The method relies on a projective approximation of the exact cavity distributions of imaginary time trajectories (paths). A key feature, novel in the context of similar algorithms, is the explicit separation of the classical and quantum parts of the distributions. Numerical sim… ▽ More In this paper we introduce an approximate method to solve the quantum cavity equations for transverse field Ising models. The method relies on a projective approximation of the exact cavity distributions of imaginary time trajectories (paths). A key feature, novel in the context of similar algorithms, is the explicit separation of the classical and quantum parts of the distributions. Numerical simulations show accurate results in comparison with the sampled solution of the cavity equations, the exact diagonalization of the Hamiltonian (when possible) and other approximate inference methods in the literature. The computational complexity of this new algorithm scales linearly with the connectivity of the underlying lattice, enabling the study of highly connected networks, as the ones often encountered in quantum machine learning problems. △ Less

Submitted 27 January, 2023; v1 submitted 20 October, 2022; originally announced October 2022.

Adaptive Smoothing Path Integral Control

Authors: Dominik Thalmeier, Hilbert J. Kappen, Simone Totaro, Vicenç Gómez

Abstract: In Path Integral control problems a representation of an optimally controlled dynamical system can be formally computed and serve as a guidepost to learn a parametrized policy. The Path Integral Cross-Entropy (PICE) method tries to exploit this, but is hampered by poor sample efficiency. We propose a model-free algorithm called ASPIC (Adaptive Smoothing of Path Integral Control) that applies an in… ▽ More In Path Integral control problems a representation of an optimally controlled dynamical system can be formally computed and serve as a guidepost to learn a parametrized policy. The Path Integral Cross-Entropy (PICE) method tries to exploit this, but is hampered by poor sample efficiency. We propose a model-free algorithm called ASPIC (Adaptive Smoothing of Path Integral Control) that applies an inf-convolution to the cost function to speedup convergence of policy optimization. We identify PICE as the infinite smoothing limit of such technique and show that the sample efficiency problems that PICE suffers disappear for finite levels of smoothing. For zero smoothing this method becomes a greedy optimization of the cost, which is the standard approach in current reinforcement learning. We show analytically and empirically that intermediate levels of smoothing are optimal, which renders the new method superior to both PICE and direct cost-optimization. △ Less

Submitted 13 May, 2020; originally announced May 2020.

Comments: 23 pages, 5 figures, NeurIPS 2019 Optimization Foundations of Reinforcement Learning Workshop (OptRL 2019)

An atomic Boltzmann machine capable of on-chip learning

Authors: Brian Kiraly, Elze J. Knol, Hilbert J. Kappen, Alexander A. Khajetoorians

Abstract: The Boltzmann Machine (BM) is a neural network composed of stochastically firing neurons that can learn complex probability distributions by adapting the synaptic interactions between the neurons. BMs represent a very generic class of stochastic neural networks that can be used for data clustering, generative modelling and deep learning. A key drawback of software-based stochastic neural networks… ▽ More The Boltzmann Machine (BM) is a neural network composed of stochastically firing neurons that can learn complex probability distributions by adapting the synaptic interactions between the neurons. BMs represent a very generic class of stochastic neural networks that can be used for data clustering, generative modelling and deep learning. A key drawback of software-based stochastic neural networks is the required Monte Carlo sampling, which scales intractably with the number of neurons. Here, we realize a physical implementation of a BM directly in the stochastic spin dynamics of a gated ensemble of coupled cobalt atoms on the surface of semiconducting black phosphorus. Implementing the concept of orbital memory utilizing scanning tunnelling microscopy, we demonstrate the bottom-up construction of atomic ensembles whose stochastic current noise is defined by a reconfigurable multi-well energy landscape. Exploiting the anisotropic behaviour of black phosphorus, we build ensembles of atoms with two well-separated intrinsic time scales that represent neurons and synapses. By characterizing the conditional steady-state distribution of the neurons for given synaptic configurations, we illustrate that an ensemble can represent many distinct probability distributions. By probing the intrinsic synaptic dynamics, we reveal an autonomous reorganization of the synapses in response to external electrical stimuli. This self-adaptive architecture paves the way for on-chip learning directly in atomic-scale machine learning hardware. △ Less

Submitted 15 September, 2021; v1 submitted 4 May, 2020; originally announced May 2020.

Journal ref: Nature Nanotechnology, 16, 414 (2021)

Decision Support in the Context of a Complex Decision Situation

Authors: Teus H. Kappen, Mirko Noordegraaf, Wilton A. van Klei, Karel G. M. Moons, Cor J. Kalkman

Abstract: The aim of a clinical decision support tool is to reduce the complexity of clinical decisions. However, when decision support tools are poorly implemented they may actually confuse physicians and complicate clinical care. This paper argues that information from decision support tools is often removed from the clinical context of the targeted decisions. Physicians largely depend on clinical context… ▽ More The aim of a clinical decision support tool is to reduce the complexity of clinical decisions. However, when decision support tools are poorly implemented they may actually confuse physicians and complicate clinical care. This paper argues that information from decision support tools is often removed from the clinical context of the targeted decisions. Physicians largely depend on clinical context to handle the complexity of their day-to-day decisions. Clinical context enables them to take into account all ambiguous information and patient preferences. Decision support tools that provide analytic information to physicians, without its context, may then complicate the decision process of physicians. It is likely that the joint forces of physicians and technology will produce better decisions than either of them exclusively: after all, they do have different ways of dealing with the complexity of a decision and are thus complementary. Therefore, the future challenges of decision support do not only reside in the optimization of the predictive value of the underlying models and algorithms, but equally in the effective communication of information and its context to doctors. △ Less

Submitted 2 March, 2020; originally announced March 2020.

Implementing perceptron models with qubits

Authors: Roeland Wiersema, H. J. Kappen

Abstract: We propose a method for learning a quantum probabilistic model of a perceptron. By considering a cross entropy between two density matrices we can learn a model that takes noisy output labels into account while learning. A multitude of proposals already exist that aim to utilize the curious properties of quantum systems to build a quantum perceptron, but these proposals rely on a classical cost fu… ▽ More We propose a method for learning a quantum probabilistic model of a perceptron. By considering a cross entropy between two density matrices we can learn a model that takes noisy output labels into account while learning. A multitude of proposals already exist that aim to utilize the curious properties of quantum systems to build a quantum perceptron, but these proposals rely on a classical cost function for the optimization procedure. We demonstrate the usage of a quantum equivalent of the classical log-likelihood, which allows for a quantum model and training procedure. We show that this allows us to better capture noisyness in data compared to a classical perceptron. By considering entangled qubits we can learn nonlinear separation boundaries, such as XOR. △ Less

Submitted 22 August, 2019; v1 submitted 16 May, 2019; originally announced May 2019.

Journal ref: Phys. Rev. A 100, 020301 (2019)

Atom-by-atom construction of attractors in a tunable finite size spin array

Authors: Alex Kolmus, Mikhail I. Katsnelson, Alexander A. Khajetoorians, Hilbert J. Kappen

Abstract: We demonstrate that a two-dimensional finite and periodic array of Ising spins coupled via RKKY-like exchange can exhibit tunable magnetic states ranging from three distinct magnetic regimes: (1) a conventional ferromagnetic regime, (2) a glass-like regime, and (3) a new multi-well regime. These magnetic regimes can be tuned by one gate-like parameter, namely the ratio between the lattice constant… ▽ More We demonstrate that a two-dimensional finite and periodic array of Ising spins coupled via RKKY-like exchange can exhibit tunable magnetic states ranging from three distinct magnetic regimes: (1) a conventional ferromagnetic regime, (2) a glass-like regime, and (3) a new multi-well regime. These magnetic regimes can be tuned by one gate-like parameter, namely the ratio between the lattice constant and the oscillating interaction wavelength. We characterize the various magnetic regimes, quantifying the distribution of low energy states, aging relaxation dynamics, and scaling behavior. The glassy and multi-well behavior results from the competing character of the oscillating long-range exchange interactions. The multi-well structure features multiple attractors, each with a sizable basin of attraction. This may open the possible application of such atomic arrays as associative memories. △ Less

Submitted 30 October, 2019; v1 submitted 22 April, 2018; originally announced April 2018.

Journal ref: New J. Phys. 22 023038 (2020)

Learning quantum models from quantum or classical data

Authors: Hilbert J Kappen

Abstract: In this paper, we address the problem how to represent a classical data distribution in a quantum system. The proposed method is to learn quantum Hamiltonian that is such that its ground state approximates the given classical distribution. We review previous work on the quantum Boltzmann machine (QBM) and how it can be used to infer quantum Hamiltonians from quantum statistics. We then show how th… ▽ More In this paper, we address the problem how to represent a classical data distribution in a quantum system. The proposed method is to learn quantum Hamiltonian that is such that its ground state approximates the given classical distribution. We review previous work on the quantum Boltzmann machine (QBM) and how it can be used to infer quantum Hamiltonians from quantum statistics. We then show how the proposed quantum learning formalism can also be applied to a purely classical data analysis. Representing the data as a rank one density matrix introduces quantum statistics for classical data in addition to the classical statistics. We show that quantum learning yields results that can be significantly more accurate than the classical maximum likelihood approach, both for unsupervised learning and for classification. The data density matrix and the QBM solution show entanglement, quantified by the quantum mutual information $I$. The classical mutual information in the data $I_c\le I/2=C$, with $C$ maximal classical correlations obtained by choosing a suitable orthogonal measurement basis. We suggest that the remaining mutual information $Q=I/2$ is obtained by non orthogonal measurements that may violate the Bell inequality. The excess mutual information $I-I_c$ may potentially be used to improve the performance of quantum implementations of machine learning or other statistical methods. △ Less

Submitted 15 January, 2020; v1 submitted 29 March, 2018; originally announced March 2018.

Comments: 28 pages, 7 figures

Nonlinear Deconvolution by Sampling Biophysically Plausible Hemodynamic Models

Authors: Hans-Christian Ruiz-Euler, Jose R. Ferreira Marques, Hilbert J. Kappen

Abstract: Non-invasive methods to measure brain activity are important to understand cognitive processes in the human brain. A prominent example is functional magnetic resonance imaging (fMRI), which is a noisy measurement of a delayed signal that depends non-linearly on the neuronal activity through the neurovascular coupling. These characteristics make the inference of neuronal activity from fMRI a diffic… ▽ More Non-invasive methods to measure brain activity are important to understand cognitive processes in the human brain. A prominent example is functional magnetic resonance imaging (fMRI), which is a noisy measurement of a delayed signal that depends non-linearly on the neuronal activity through the neurovascular coupling. These characteristics make the inference of neuronal activity from fMRI a difficult but important step in fMRI studies that require information at the neuronal level. In this article, we address this inference problem using a Bayesian approach where we model the latent neural activity as a stochastic process and assume that the observed BOLD signal results from a realistic physiological (Balloon) model. We apply a recently developed smoothing method called APIS to efficiently sample the posterior given single event fMRI time series. To infer neuronal signals with high likelihood for multiple time series efficiently, a modification of the original algorithm is introduced. We demonstrate that our adaptive procedure is able to compensate the lacking of inputs in the model to infer the neuronal activity and that it outperforms dramatically the standard bootstrap particle filter-smoother in this setting. This makes the proposed procedure especially attractive to deconvolve resting state fMRI data. To validate the method, we evaluate the quality of the signals inferred using the timing information contained in them. APIS obtains reliable event timing estimates based on fMRI data gathered during a reaction time experiment with short stimuli. Hence, we show for the first time that one can obtain accurate absolute timing of neuronal activity by reconstructing the latent neural signal. △ Less

Submitted 23 March, 2018; originally announced March 2018.

Consistent Adaptive Multiple Importance Sampling and Controlled Diffusions

Authors: Sep Thijssen, H. J. Kappen

Abstract: Recent progress has been made with Adaptive Multiple Importance Sampling (AMIS) methods that show improvement in effective sample size. However, consistency for the AMIS estimator has only been established in very restricted cases. Furthermore, the high computational complexity of the re-weighting in AMIS (called balance heuristic) makes it expensive for applications involving diffusion processes.… ▽ More Recent progress has been made with Adaptive Multiple Importance Sampling (AMIS) methods that show improvement in effective sample size. However, consistency for the AMIS estimator has only been established in very restricted cases. Furthermore, the high computational complexity of the re-weighting in AMIS (called balance heuristic) makes it expensive for applications involving diffusion processes. In this work we consider sequential and adaptive importance sampling that is particularly suitable for diffusion processes. We propose a new discarding-re-weighting scheme that is of lower computational complexity, and we prove that the resulting AMIS is consistent. Using numerical experiments, we demonstrate that discarding-re-weighting performs very similar to the balance heuristic, but at a fraction of the computational cost. △ Less

Submitted 21 March, 2018; originally announced March 2018.

Effective Connectivity from Single Trial fMRI Data by Sampling Biologically Plausible Models

Authors: H. C. Ruiz-Euler, H. J. Kappen

Abstract: The estimation of causal network architectures in the brain is fundamental for understanding cognitive information processes. However, access to the dynamic processes underlying cognition is limited to indirect measurements of the hidden neuronal activity, for instance through fMRI data. Thus, estimating the network structure of the underlying process is challenging. In this article, we embed an a… ▽ More The estimation of causal network architectures in the brain is fundamental for understanding cognitive information processes. However, access to the dynamic processes underlying cognition is limited to indirect measurements of the hidden neuronal activity, for instance through fMRI data. Thus, estimating the network structure of the underlying process is challenging. In this article, we embed an adaptive importance sampler called Adaptive Path Integral Smoother (APIS) into the Expectation-Maximization algorithm to obtain point estimates of causal connectivity. We demonstrate on synthetic data that this procedure finds not only the correct network structure but also the direction of effective connections from random initializations of the connectivity matrix. In addition--motivated by contradictory claims in the literature--we examine the effect of the neuronal timescale on the sensitivity of the BOLD signal to changes in the connectivity and on the maximum likelihood solutions of the connectivity. We conclude with two warnings: First, the connectivity estimates under the assumption of slow dynamics can be extremely biased if the data was generated by fast neuronal processes. Second, the faster the time scale, the less sensitive the BOLD signal is to changes in the incoming connections to a node. Hence, connectivity estimation using realistic neural dynamics timescale requires extremely high-quality data and seems infeasible in many practical data sets. △ Less

Submitted 15 March, 2018; originally announced March 2018.

On the role of synaptic stochasticity in training low-precision neural networks

Authors: Carlo Baldassi, Federica Gerace, Hilbert J. Kappen, Carlo Lucibello, Luca Saglietti, Enzo Tartaglione, Riccardo Zecchina

Abstract: Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performanc… ▽ More Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension aimed at training discrete deep neural networks is also investigated. △ Less

Submitted 19 March, 2018; v1 submitted 26 October, 2017; originally announced October 2017.

Comments: 7 pages + 14 pages of supplementary material

Journal ref: Phys. Rev. Lett. 120, 268103 (2018)

Efficient Optical flow and Stereo Vision for Velocity Estimation and Obstacle Avoidance on an Autonomous Pocket Drone

Authors: Kimberly McGuire, Guido de Croon, Christophe De Wagter, Karl Tuyls, Hilbert Kappen

Abstract: Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and… ▽ More Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and uses feature histograms to calculate optical flow and stereo disparity. The stereo-based distance estimates are used to scale the optical flow in order to retrieve the drone's velocity. The velocity and depth measurements are used for fully autonomous flight of a 40 g pocket drone only relying on on-board sensors. The method allows the MAV to control its velocity and avoid obstacles. △ Less

Submitted 14 March, 2017; v1 submitted 20 December, 2016; originally announced December 2016.

Comments: 7 pages, 10 figures, Published at IEEE Robotics and Automation Letters

Journal ref: IEEE Robotics and Automation Letters, 2017, 2, 1070-1076

Action selection in growing state spaces: Control of Network Structure Growth

Authors: Dominik Thalmeier, Vicenç Gómez, Hilbert J. Kappen

Abstract: The dynamical processes taking place on a network depend on its topology. Influencing the growth process of a network therefore has important implications on such dynamical processes. We formulate the problem of influencing the growth of a network as a stochastic optimal control problem in which a structural cost function penalizes undesired topologies. We approximate this control problem with a r… ▽ More The dynamical processes taking place on a network depend on its topology. Influencing the growth process of a network therefore has important implications on such dynamical processes. We formulate the problem of influencing the growth of a network as a stochastic optimal control problem in which a structural cost function penalizes undesired topologies. We approximate this control problem with a restricted class of control problems that can be solved using probabilistic inference methods. To deal with the increasing problem dimensionality, we introduce an adaptive importance sampling method for approximating the optimal control. We illustrate this methodology in the context of formation of information cascades, considering the task of influencing the structure of a growing conversation thread, as in Internet forums. Using a realistic model of growing trees, we show that our approach can yield conversation threads with better structural properties than the ones observed without control. △ Less

Submitted 27 December, 2016; v1 submitted 23 June, 2016; originally announced June 2016.

Comments: 23 pages, 7 figures

Journal ref: Journal of Physics A: Mathematical and Theoretical, Volume 50, Number 3, 034006, 2017

Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother

Authors: H. -Ch. Ruiz, H. J. Kappen

Abstract: Particle smoothing methods are used for inference of stochastic processes based on noisy observations. Typically, the estimation of the marginal posterior distribution given all observations is cumbersome and computational intensive. In this paper, we propose a simple algorithm based on path integral control theory to estimate the smoothing distribution of continuous-time diffusion processes with… ▽ More Particle smoothing methods are used for inference of stochastic processes based on noisy observations. Typically, the estimation of the marginal posterior distribution given all observations is cumbersome and computational intensive. In this paper, we propose a simple algorithm based on path integral control theory to estimate the smoothing distribution of continuous-time diffusion processes with partial observations. In particular, we use an adaptive importance sampling method to improve the effective sampling size of the posterior over processes given the observations and the reliability of the estimation of the marginals. This is achieved by estimating a feedback controller to sample efficiently from the joint smoothing distributions. We compare the results with estimations obtained from the standard Forward Filter/Backward Simulator for two diffusion processes of different complexity. We show that the proposed method gives more reliable estimations than the standard FFBSi when the smoothing distribution is poorly represented by the filter distribution. △ Less

Submitted 6 March, 2017; v1 submitted 1 May, 2016; originally announced May 2016.

Comments: 16 pages, 13 figures

Local Histogram Matching for Efficient Optical Flow Computation Applied to Velocity Estimation on Pocket Drones

Authors: Kimberly McGuire, Guido de Croon, Christophe de Wagter, Bart Remes, Karl Tuyls, Hilbert Kappen

Abstract: Autonomous flight of pocket drones is challenging due to the severe limitations on on-board energy, sensing, and processing power. However, tiny drones have great potential as their small size allows maneuvering through narrow spaces while their small weight provides significant safety advantages. This paper presents a computationally efficient algorithm for determining optical flow, which can be… ▽ More Autonomous flight of pocket drones is challenging due to the severe limitations on on-board energy, sensing, and processing power. However, tiny drones have great potential as their small size allows maneuvering through narrow spaces while their small weight provides significant safety advantages. This paper presents a computationally efficient algorithm for determining optical flow, which can be run on an STM32F4 microprocessor (168 MHz) of a 4 gram stereo-camera. The optical flow algorithm is based on edge histograms. We propose a matching scheme to determine local optical flow. Moreover, the method allows for sub-pixel flow determination based on time horizon adaptation. We demonstrate velocity measurements in flight and use it within a velocity control-loop on a pocket drone. △ Less

Submitted 14 March, 2017; v1 submitted 24 March, 2016; originally announced March 2016.

Comments: 7 pages, 10 figures, Changes: format changed one column to two columns, used url package for links

Journal ref: 2016 IEEE International Conference on Robotics and Automation (ICRA), 3255 - 3260,

Learning universal computations with spikes

Authors: Dominik Thalmeier, Marvin Uhlmann, Hilbert J. Kappen, Raoul-Martin Memmesheimer

Abstract: Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g.~for locomotion. Many such computations require previous building of intrinsic world mod… ▽ More Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g.~for locomotion. Many such computations require previous building of intrinsic world models. Here we show how spiking neural networks may solve these different tasks. Firstly, we derive constraints under which classes of spiking neural networks lend themselves to substrates of powerful general purpose computing. The networks contain dendritic or synaptic nonlinearities and have a constrained connectivity. We then combine such networks with learning rules for outputs or recurrent connections. We show that this allows to learn even difficult benchmark tasks such as the self-sustained generation of desired low-dimensional chaotic dynamics or memory-dependent computations. Furthermore, we show how spiking networks can build models of external world systems and use the acquired knowledge to control them. △ Less

Submitted 29 June, 2016; v1 submitted 28 May, 2015; originally announced May 2015.

Journal ref: PLoS Comput Biol 12(6): e1004895 (2016)

Adaptive importance sampling for control and inference

Authors: Hilbert Johan Kappen, Hans Christian Ruiz

Abstract: Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feyman-Kac path integral and can be estimated using Monte Carlo sampling. In this contribution we review path integral control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. Within the PI theory, the qu… ▽ More Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feyman-Kac path integral and can be estimated using Monte Carlo sampling. In this contribution we review path integral control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the Path Integral Cross Entropy method or PICE. We illustrate this method for some simple examples. The path integral control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the path integral control method as an accurate alternative to particle filtering. △ Less

Submitted 2 September, 2015; v1 submitted 7 May, 2015; originally announced May 2015.

Comments: 23 pages, 4 figures

Real-Time Stochastic Optimal Control for Multi-agent Quadrotor Systems

Authors: Vicenç Gómez, Sep Thijssen, Andrew Symington, Stephen Hailes, Hilbert J. Kappen

Abstract: This paper presents a novel method for controlling teams of unmanned aerial vehicles using Stochastic Optimal Control (SOC) theory. The approach consists of a centralized high-level planner that computes optimal state trajectories as velocity sequences, and a platform-specific low-level controller which ensures that these velocity sequences are met. The planning task is expressed as a centralized… ▽ More This paper presents a novel method for controlling teams of unmanned aerial vehicles using Stochastic Optimal Control (SOC) theory. The approach consists of a centralized high-level planner that computes optimal state trajectories as velocity sequences, and a platform-specific low-level controller which ensures that these velocity sequences are met. The planning task is expressed as a centralized path-integral control problem, for which optimal control computation corresponds to a probabilistic inference problem that can be solved by efficient sampling methods. Through simulation we show that our SOC approach (a) has significant benefits compared to deterministic control and other SOC methods in multimodal problems with noise-dependent optimal solutions, (b) is capable of controlling a large number of platforms in real-time, and (c) yields collective emergent behaviour in the form of flight formations. Finally, we show that our approach works for real platforms, by controlling a team of three quadrotors in outdoor conditions. △ Less

Submitted 12 May, 2020; v1 submitted 16 February, 2015; originally announced February 2015.

Comments: 17 pages, 8 figures, 26th International Conference on Automated Planning and Scheduling

Approximate inference on planar graphs using Loop Calculus and Belief Propagation

Authors: Vicenc Gomez, Hilbert Kappen, Michael Chertkov

Abstract: We introduce novel results for approximate inference on planar graphical models using the loop calculus framework. The loop calculus (Chertkov and Chernyak, 2006b) allows to express the exact partition function Z of a graphical model as a finite sum of terms that can be evaluated once the belief propagation (BP) solution is known. In general, full summation over all correction terms is intractable… ▽ More We introduce novel results for approximate inference on planar graphical models using the loop calculus framework. The loop calculus (Chertkov and Chernyak, 2006b) allows to express the exact partition function Z of a graphical model as a finite sum of terms that can be evaluated once the belief propagation (BP) solution is known. In general, full summation over all correction terms is intractable. We develop an algorithm for the approach presented in Chertkov et al. (2008) which represents an efficient truncation scheme on planar graphs and a new representation of the series in terms of Pfaffians of matrices. We analyze in detail both the loop series and the Pfaffian series for models with binary variables and pairwise interactions, and show that the first term of the Pfaffian series can provide very accurate approximations. The algorithm outperforms previous truncation schemes of the loop series and is competitive with other state-of-the-art methods for approximate inference. △ Less

Submitted 9 August, 2014; originally announced August 2014.

Comments: Appears in Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI2009)

Report number: UAI-P-2009-PG-195-202

Path Integral Control and State Dependent Feedback

Authors: Sep Thijssen, H. J. Kappen

Abstract: In this paper we address the problem to compute state dependent feedback controls for path integral control problems. To this end we generalize the path integral control formula and utilize this to construct parameterized state dependent feedback controllers. In addition, we show a novel relation between control and importance sampling: better control, in terms of control cost, yields more efficie… ▽ More In this paper we address the problem to compute state dependent feedback controls for path integral control problems. To this end we generalize the path integral control formula and utilize this to construct parameterized state dependent feedback controllers. In addition, we show a novel relation between control and importance sampling: better control, in terms of control cost, yields more efficient importance sampling, in terms of effective sample size. The optimal control provides a zero-variance estimate. △ Less

Submitted 5 January, 2016; v1 submitted 16 June, 2014; originally announced June 2014.

Latent Kullback Leibler Control for Continuous-State Systems using Probabilistic Graphical Models

Authors: Takamitsu Matsubara, Vicenç Gómez, Hilbert J. Kappen

Abstract: Kullback Leibler (KL) control problems allow for efficient computation of optimal control by solving a principal eigenvector problem. However, direct applicability of such framework to continuous state-action systems is limited. In this paper, we propose to embed a KL control problem in a probabilistic graphical model where observed variables correspond to the continuous (possibly high-dimensional… ▽ More Kullback Leibler (KL) control problems allow for efficient computation of optimal control by solving a principal eigenvector problem. However, direct applicability of such framework to continuous state-action systems is limited. In this paper, we propose to embed a KL control problem in a probabilistic graphical model where observed variables correspond to the continuous (possibly high-dimensional) state of the system and latent variables correspond to a discrete (low-dimensional) representation of the state amenable for KL control computation. We present two examples of this approach. The first one uses standard hidden Markov models (HMMs) and computes exact optimal control, but is only applicable to low-dimensional systems. The second one uses factorial HMMs, it is scalable to higher dimensional problems, but control computation is approximate. We illustrate both examples in several robot motor control tasks. △ Less

Submitted 27 August, 2014; v1 submitted 4 June, 2014; originally announced June 2014.

Comments: 9 pages, 5 figures, accepted in Uncertainty in Artificial Intelligence (UAI '14)

ACM Class: I.2.8; I.2.9; G.3

Comment: Causal entropic forces

Authors: H. J. Kappen

Abstract: In this comment I argue that the causal entropy proposed in [1] is state-independent and the entropic force is zero for state-independent noise in a discrete time formulation and that the causal entropy description is incomplete in the continuous time case. In this comment I argue that the causal entropy proposed in [1] is state-independent and the entropic force is zero for state-independent noise in a discrete time formulation and that the causal entropy description is incomplete in the continuous time case. △ Less

Submitted 15 December, 2013; originally announced December 2013.

Comments: 2 pages, no figures

Stochastic Optimal Control as Non-equilibrium Statistical Mechanics: Calculus of Variations over Density and Current

Authors: Vladimir Y. Chernyak, Michael Chertkov, Joris Bierkens, Hilbert J. Kappen

Abstract: In Stochastic Optimal Control (SOC) one minimizes the average cost-to-go, that consists of the cost-of-control (amount of efforts), cost-of-space (where one wants the system to be) and the target cost (where one wants the system to arrive), for a system participating in forced and controlled Langevin dynamics. We extend the SOC problem by introducing an additional cost-of-dynamics, characterized b… ▽ More In Stochastic Optimal Control (SOC) one minimizes the average cost-to-go, that consists of the cost-of-control (amount of efforts), cost-of-space (where one wants the system to be) and the target cost (where one wants the system to arrive), for a system participating in forced and controlled Langevin dynamics. We extend the SOC problem by introducing an additional cost-of-dynamics, characterized by a vector potential. We propose derivation of the generalized gauge-invariant Hamilton-Jacobi-Bellman equation as a variation over density and current, suggest hydrodynamic interpretation and discuss examples, e.g., ergodic control of a particle-within-a-circle, illustrating non-equilibrium space-time complexity. △ Less

Submitted 27 June, 2013; originally announced June 2013.

Comments: 4 pages, 1 figure

Journal ref: 2013 J. Phys. A: Math. Theor. 47

Linear PDEs and eigenvalue problems corresponding to ergodic stochastic optimization problems on compact manifolds

Authors: Joris Bierkens, Vladimir Y. Chernyak, Michael Chertkov, Hilbert J. Kappen

Abstract: We consider long term average or `ergodic' optimal control poblems with a special structure: Control is exerted in all directions and the control costs are proportional to the square of the norm of the control field with respect to the metric induced by the noise. The long term stochastic dynamics on the manifold will be completely characterized by the long term density $ρ$ and the long term curre… ▽ More We consider long term average or `ergodic' optimal control poblems with a special structure: Control is exerted in all directions and the control costs are proportional to the square of the norm of the control field with respect to the metric induced by the noise. The long term stochastic dynamics on the manifold will be completely characterized by the long term density $ρ$ and the long term current density $J$. As such, control problems may be reformulated as variational problems over $ρ$ and $J$. We discuss several optimization problems: the problem in which both $ρ$ and $J$ are varied freely, the problem in which $ρ$ is fixed and the one in which $J$ is fixed. These problems lead to different kinds of operator problems: linear PDEs in the first two cases and a nonlinear PDE in the latter case. These results are obtained through through variational principle using infinite dimensional Lagrange multipliers. In the case where the initial dynamics are reversible we obtain the result that the optimally controlled diffusion is also symmetrizable. The particular case of constraining the dynamics to be reversible of the optimally controlled process leads to a linear eigenvalue problem for the square root of the density process. △ Less

Submitted 29 January, 2016; v1 submitted 1 March, 2013; originally announced March 2013.

MSC Class: 49K20; 93E20; 58A25

Journal ref: Journal of Statistical Mechanics: Theory and Experiment, 2016(1), 13206

General Lower Bounds based on Computer Generated Higher Order Expansions

Authors: Martijn Leisink, Hilbert Kappen

Abstract: In this article we show the rough outline of a computer algorithm to generate lower bounds on the exponential function of (in principle) arbitrary precision. We implemented this to generate all necessary analytic terms for the Boltzmann machine partition function thus leading to lower bounds of any order. It turns out that the extra variational parameters can be optimized analytically. We show… ▽ More In this article we show the rough outline of a computer algorithm to generate lower bounds on the exponential function of (in principle) arbitrary precision. We implemented this to generate all necessary analytic terms for the Boltzmann machine partition function thus leading to lower bounds of any order. It turns out that the extra variational parameters can be optimized analytically. We show that bounds upto nineth order are still reasonably calculable in practical situations. The generated terms can also be used as extra correction terms (beyond TAP) in mean field expansions. △ Less

Submitted 12 December, 2012; originally announced January 2013.

Comments: Appears in Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence (UAI2002)

Report number: UAI-P-2002-PG-293-300

Approximate Inference and Constrained Optimization

Authors: Tom Heskes, Kees Albers, Hilbert Kappen

Abstract: Loopy and generalized belief propagation are popular algorithms for approximate inference in Markov random fields and Bayesian networks. Fixed points of these algorithms correspond to extrema of the Bethe and Kikuchi free energy. However, belief propagation does not always converge, which explains the need for approaches that explicitly minimize the Kikuchi/Bethe free energy, such… ▽ More Loopy and generalized belief propagation are popular algorithms for approximate inference in Markov random fields and Bayesian networks. Fixed points of these algorithms correspond to extrema of the Bethe and Kikuchi free energy. However, belief propagation does not always converge, which explains the need for approaches that explicitly minimize the Kikuchi/Bethe free energy, such as CCCP and UPS. Here we describe a class of algorithms that solves this typically nonconvex constrained minimization of the Kikuchi free energy through a sequence of convex constrained minimizations of upper bounds on the Kikuchi free energy. Intuitively one would expect tighter bounds to lead to faster algorithms, which is indeed convincingly demonstrated in our simulations. Several ideas are applied to obtain tight convex bounds that yield dramatic speed-ups over CCCP. △ Less

Submitted 19 October, 2012; originally announced December 2012.

Comments: Appears in Proceedings of the Nineteenth Conference on Uncertainty in Artificial Intelligence (UAI2003)

Report number: UAI-P-2003-PG-313-320

Learning Price-Elasticity of Smart Consumers in Power Distribution Systems

Authors: Vicenç Gómez, Michael Chertkov, Scott Backhaus, Hilbert J. Kappen

Abstract: Demand Response is an emerging technology which will transform the power grid of tomorrow. It is revolutionary, not only because it will enable peak load shaving and will add resources to manage large distribution systems, but mainly because it will tap into an almost unexplored and extremely powerful pool of resources comprised of many small individual consumers on distribution grids. However, to… ▽ More Demand Response is an emerging technology which will transform the power grid of tomorrow. It is revolutionary, not only because it will enable peak load shaving and will add resources to manage large distribution systems, but mainly because it will tap into an almost unexplored and extremely powerful pool of resources comprised of many small individual consumers on distribution grids. However, to utilize these resources effectively, the methods used to engage these resources must yield accurate and reliable control. A diversity of methods have been proposed to engage these new resources. As opposed to direct load control, many methods rely on consumers and/or loads responding to exogenous signals, typically in the form of energy pricing, originating from the utility or system operator. Here, we propose an open loop communication-lite method for estimating the price elasticity of many customers comprising a distribution system. We utilize a sparse linear regression method that relies on operator-controlled, inhomogeneous minor price variations, which will be fair to all the consumers. Our numerical experiments show that reliable estimation of individual and thus aggregated instantaneous elasticities is possible. We describe the limits of the reliable reconstruction as functions of the three key parameters of the system: (i) ratio of the number of communication slots (time units) per number of engaged consumers; (ii) level of sparsity (in consumer response); and (iii) signal-to-noise ratio. △ Less

Submitted 25 September, 2012; originally announced September 2012.

Comments: 6 pages, 5 figures, IEEE SmartGridComm 2012

ACM Class: C.2.1; G.3

Sufficient conditions for convergence of Loopy Belief Propagation

Authors: Joris Mooij, Hilbert Kappen

Abstract: We derive novel sufficient conditions for convergence of Loopy Belief Propagation (also known as the Sum-Product algorithm) to a unique fixed point. Our results improve upon previously known conditions. For binary variables with (anti-)ferromagnetic interactions, our conditions seem to be sharp. We derive novel sufficient conditions for convergence of Loopy Belief Propagation (also known as the Sum-Product algorithm) to a unique fixed point. Our results improve upon previously known conditions. For binary variables with (anti-)ferromagnetic interactions, our conditions seem to be sharp. △ Less

Submitted 4 July, 2012; originally announced July 2012.

Comments: Appears in Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI2005)

Report number: UAI-P-2005-PG-396-403

Stochastic Optimal Control in Continuous Space-Time Multi-Agent Systems

Authors: Wim Wiegerinck, Bart van den Broek, Hilbert Kappen

Abstract: Recently, a theory for stochastic optimal control in non-linear dynamical systems in continuous space-time has been developed (Kappen, 2005). We apply this theory to collaborative multi-agent systems. The agents evolve according to a given non-linear dynamics with additive Wiener noise. Each agent can control its own dynamics. The goal is to minimize the accumulated joint cost, which consists of a… ▽ More Recently, a theory for stochastic optimal control in non-linear dynamical systems in continuous space-time has been developed (Kappen, 2005). We apply this theory to collaborative multi-agent systems. The agents evolve according to a given non-linear dynamics with additive Wiener noise. Each agent can control its own dynamics. The goal is to minimize the accumulated joint cost, which consists of a state dependent term and a term that is quadratic in the control. We focus on systems of non-interacting agents that have to distribute themselves optimally over a number of targets, given a set of end-costs for the different possible agent-target combinations. We show that optimal control is the combinatorial sum of independent single-agent single-target optimal controls weighted by a factor proportional to the end-costs of the different combinations. Thus, multi-agent control is related to a standard graphical model inference problem. The additional computational cost compared to single-agent control is exponential in the tree-width of the graph specifying the combinatorial sum times the number of targets. We illustrate the result by simulations of systems with up to 42 agents. △ Less

Submitted 27 June, 2012; originally announced June 2012.

Comments: Appears in Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence (UAI2006)

Report number: UAI-P-2006-PG-528-535

Hybrid Variational/Gibbs Collapsed Inference in Topic Models

Authors: Max Welling, Yee Whye Teh, Hilbert Kappen

Abstract: Variational Bayesian inference and (collapsed) Gibbs sampling are the two important classes of inference algorithms for Bayesian networks. Both have their advantages and disadvantages: collapsed Gibbs sampling is unbiased but is also inefficient for large count values and requires averaging over many samples to reduce variance. On the other hand, variational Bayesian inference is efficient and acc… ▽ More Variational Bayesian inference and (collapsed) Gibbs sampling are the two important classes of inference algorithms for Bayesian networks. Both have their advantages and disadvantages: collapsed Gibbs sampling is unbiased but is also inefficient for large count values and requires averaging over many samples to reduce variance. On the other hand, variational Bayesian inference is efficient and accurate for large count values but suffers from bias for small counts. We propose a hybrid algorithm that combines the best of both worlds: it samples very small counts and applies variational updates to large counts. This hybridization is shown to significantly improve testset perplexity relative to variational inference at no computational cost. △ Less

Submitted 13 June, 2012; originally announced June 2012.

Comments: Appears in Proceedings of the Twenty-Fourth Conference on Uncertainty in Artificial Intelligence (UAI2008)

Report number: UAI-P-2008-PG-587-594

Explicit solution of relative entropy weighted control

Authors: Joris Bierkens, Hilbert J. Kappen

Abstract: We consider the minimization over probability measures of the expected value of a random variable, regularized by relative entropy with respect to a given probability distribution. In the general setting we provide a complete characterization of the situations in which a finite optimal value exists and the situations in which a minimizing probability distribution exists. Specializing to the case w… ▽ More We consider the minimization over probability measures of the expected value of a random variable, regularized by relative entropy with respect to a given probability distribution. In the general setting we provide a complete characterization of the situations in which a finite optimal value exists and the situations in which a minimizing probability distribution exists. Specializing to the case where the underlying probability distribution is Wiener measure, we characterize finite relative entropy changes of measure in terms of square integrability of the corresponding change of drift. For the optimal change of measure for the relative entropy weighted optimization, an expression involving the Malliavin derivative of the cost random variable is derived. The theory is illustrated by its application to several examples, including the case where the cost variable is the maximum of a standard Brownian motion over a finite time horizon. For this example we obtain an exact optimal drift, as well as an approximation of the optimal drift through a Monte-Carlo algorithm. △ Less

Submitted 5 August, 2014; v1 submitted 31 May, 2012; originally announced May 2012.

MSC Class: 93E20; 60H07; 94A17

Journal ref: Systems & Control Letters Volume 72, October 2014, Pages 36-43

Risk Sensitive Path Integral Control

Authors: Bart van den Broek, Wim Wiegerinck, Hilbert Kappen

Abstract: Recently path integral methods have been developed for stochastic optimal control for a wide class of models with non-linear dynamics in continuous space-time. Path integral methods find the control that minimizes the expected cost-to-go. In this paper we show that under the same assumptions, path integral methods generalize directly to risk sensitive stochastic optimal control. Here the method mi… ▽ More Recently path integral methods have been developed for stochastic optimal control for a wide class of models with non-linear dynamics in continuous space-time. Path integral methods find the control that minimizes the expected cost-to-go. In this paper we show that under the same assumptions, path integral methods generalize directly to risk sensitive stochastic optimal control. Here the method minimizes in expectation an exponentially weighted cost-to-go. Depending on the exponential weight, risk seeking or risk averse behaviour is obtained. We demonstrate the approach on risk sensitive stochastic optimal control problems beyond the linear-quadratic case, showing the intricate interaction of multi-modal control with risk sensitivity. △ Less

Submitted 15 March, 2012; originally announced March 2012.

Comments: Appears in Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI2010)

Report number: UAI-P-2010-PG-615-622

A likelihood-based framework for the analysis of discussion threads

Authors: Vicenç Gómez, Hilbert J. Kappen, Nelly Litvak, Andreas Kaltenbrunner

Abstract: Online discussion threads are conversational cascades in the form of posted messages that can be generally found in social systems that comprise many-to-many interaction such as blogs, news aggregators or bulletin board systems. We propose a framework based on generative models of growing trees to analyse the structure and evolution of discussion threads. We consider the growth of a discussion to… ▽ More Online discussion threads are conversational cascades in the form of posted messages that can be generally found in social systems that comprise many-to-many interaction such as blogs, news aggregators or bulletin board systems. We propose a framework based on generative models of growing trees to analyse the structure and evolution of discussion threads. We consider the growth of a discussion to be determined by an interplay between popularity, novelty and a trend (or bias) to reply to the thread originator. The relevance of these features is estimated using a full likelihood approach and allows to characterize the habits and communication patterns of a given platform and/or community. △ Less

Submitted 3 March, 2012; originally announced March 2012.

Comments: 31 pages, 12 figures, journal

ACM Class: G.3; H.5.4

The Variational Garrote

Authors: Hilbert J. Kappen, Vicenç Gómez

Abstract: In this paper, we present a new variational method for sparse regression using $L_0$ regularization. The variational parameters appear in the approximate model in a way that is similar to Breiman's Garrote model. We refer to this method as the variational Garrote (VG). We show that the combination of the variational approximation and $L_0$ regularization has the effect of making the problem effect… ▽ More In this paper, we present a new variational method for sparse regression using $L_0$ regularization. The variational parameters appear in the approximate model in a way that is similar to Breiman's Garrote model. We refer to this method as the variational Garrote (VG). We show that the combination of the variational approximation and $L_0$ regularization has the effect of making the problem effectively of maximal rank even when the number of samples is small compared to the number of variables. The VG is compared numerically with the Lasso method, ridge regression and the recently introduced paired mean field method (PMF) (M. Titsias & M. Lázaro-Gredilla., NIPS 2012). Numerical results show that the VG and PMF yield more accurate predictions and more accurately reconstruct the true model than the other methods. It is shown that the VG finds correct solutions when the Lasso solution is inconsistent due to large input correlations. Globally, VG is significantly faster than PMF and tends to perform better as the problems become denser and in problems with strongly correlated inputs. The naive implementation of the VG scales cubic with the number of features. By introducing Lagrange multipliers we obtain a dual formulation of the problem that scales cubic in the number of samples, but close to linear in the number of features. △ Less

Submitted 12 November, 2012; v1 submitted 2 September, 2011; originally announced September 2011.

Comments: 26 pages, 11 figures

Modeling the structure and evolution of discussion cascades

Authors: Vicenç Gómez, Hilbert J. Kappen, Andreas Kaltenbrunner

Abstract: We analyze the structure and evolution of discussion cascades in four popular websites: Slashdot, Barrapunto, Meneame and Wikipedia. Despite the big heterogeneities between these sites, a preferential attachment (PA) model with bias to the root can capture the temporal evolution of the observed trees and many of their statistical properties, namely, probability distributions of the branching facto… ▽ More We analyze the structure and evolution of discussion cascades in four popular websites: Slashdot, Barrapunto, Meneame and Wikipedia. Despite the big heterogeneities between these sites, a preferential attachment (PA) model with bias to the root can capture the temporal evolution of the observed trees and many of their statistical properties, namely, probability distributions of the branching factors (degrees), subtree sizes and certain correlations. The parameters of the model are learned efficiently using a novel maximum likelihood estimation scheme for PA and provide a figurative interpretation about the communication habits and the resulting discussion cascades on the four different websites. △ Less

Submitted 15 April, 2011; v1 submitted 2 November, 2010; originally announced November 2010.

Comments: 10 pages, 11 figures

ACM Class: J.4; G.2.2

Journal ref: 22nd ACM conference on hypertext and hypermedia (HT 2011)

Irregular dynamics in up and down cortical states

Authors: Jorge F. Mejias, Hilbert J. Kappen, Joaquin J. Torres

Abstract: Complex coherent dynamics is present in a wide variety of neural systems. A typical example is the voltage transitions between up and down states observed in cortical areas in the brain. In this work, we study this phenomenon via a biologically motivated stochastic model of up and down transitions. The model is constituted by a simple bistable rate model, where the synaptic current is modulated by… ▽ More Complex coherent dynamics is present in a wide variety of neural systems. A typical example is the voltage transitions between up and down states observed in cortical areas in the brain. In this work, we study this phenomenon via a biologically motivated stochastic model of up and down transitions. The model is constituted by a simple bistable rate model, where the synaptic current is modulated by short-term synaptic processes which introduce stochasticity and temporal correlations. A complete analysis of our model, both with mean-field approaches and numerical simulations, shows the appearance of complex transitions between high (up) and low (down) neural activity states, driven by the synaptic noise, with permanence times in the up state distributed according to a power-law. We show that the experimentally observed large fluctuation in up and down permanence times can be explained as the result of sufficiently noisy dynamical synapses with sufficiently large recovery times. Static synapses cannot account for this behavior, nor can dynamical synapses in the absence of noise. △ Less

Submitted 20 July, 2010; originally announced July 2010.

Comments: 23 pages, 8 figues

Dynamic Policy Programming

Authors: Mohammad Gheshlaghi Azar, Vicenc Gomez, Hilbert J. Kappen

Abstract: In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation error. The bounds are expressed in terms of the l\infty-norm of the average accumula… ▽ More In this paper, we propose a novel policy iteration method, called dynamic policy programming (DPP), to estimate the optimal policy in the infinite-horizon Markov decision processes. We prove the finite-iteration and asymptotic l\infty-norm performance-loss bounds for DPP in the presence of approximation/estimation error. The bounds are expressed in terms of the l\infty-norm of the average accumulated error as opposed to the l\infty-norm of the error in the case of the standard approximate value iteration (AVI) and the approximate policy iteration (API). This suggests that DPP can achieve a better performance than AVI and API since it averages out the simulation noise caused by Monte-Carlo sampling throughout the learning process. We examine this theoretical results numerically by com- paring the performance of the approximate variants of DPP with existing reinforcement learning (RL) methods on different problem domains. Our results show that, in all cases, DPP-based algorithms outperform other RL methods by a wide margin. △ Less

Submitted 6 September, 2011; v1 submitted 12 April, 2010; originally announced April 2010.

Comments: Submitted to Journal of Machine Learning Research

Approximate inference on planar graphs using Loop Calculus and Belief Propagation

Authors: V. Gómez, H. J. Kappen, M. Chertkov

Abstract: We introduce novel results for approximate inference on planar graphical models using the loop calculus framework. The loop calculus (Chertkov and Chernyak, 2006) allows to express the exact partition function of a graphical model as a finite sum of terms that can be evaluated once the belief propagation (BP) solution is known. In general, full summation over all correction terms is intractable.… ▽ More We introduce novel results for approximate inference on planar graphical models using the loop calculus framework. The loop calculus (Chertkov and Chernyak, 2006) allows to express the exact partition function of a graphical model as a finite sum of terms that can be evaluated once the belief propagation (BP) solution is known. In general, full summation over all correction terms is intractable. We develop an algorithm for the approach presented in (Certkov et al., 2008) which represents an efficient truncation scheme on planar graphs and a new representation of the series in terms of Pfaffians of matrices. We analyze the performance of the algorithm for the partition function approximation for models with binary variables and pairwise interactions on grids and other planar graphs. We study in detail both the loop series and the equivalent Pfaffian series and show that the first term of the Pfaffian series for the general, intractable planar model, can provide very accurate approximations. The algorithm outperforms previous truncation schemes of the loop series and is competitive with other state-of-the-art methods for approximate inference. △ Less

Submitted 25 May, 2009; v1 submitted 7 January, 2009; originally announced January 2009.

Comments: 23 pages, 10 figures. Submitted to Journal of Machine Learning Research. Proceedings version accepted for UAI 2009

Self-organization using synaptic plasticity

Authors: Vicenç Gómez, Andreas Kaltenbrunner, Vicente López, Hilbert J. Kappen

Abstract: Large networks of spiking neurons show abrupt changes in their collective dynamics resembling phase transitions studied in statistical physics. An example of this phenomenon is the transition from irregular, noise-driven dynamics to regular, self-sustained behavior observed in networks of integrate-and-fire neurons as the interaction strength between the neurons increases. In this work we show h… ▽ More Large networks of spiking neurons show abrupt changes in their collective dynamics resembling phase transitions studied in statistical physics. An example of this phenomenon is the transition from irregular, noise-driven dynamics to regular, self-sustained behavior observed in networks of integrate-and-fire neurons as the interaction strength between the neurons increases. In this work we show how a network of spiking neurons is able to self-organize towards a critical state for which the range of possible inter-spike-intervals (dynamic range) is maximized. Self-organization occurs via synaptic dynamics that we analytically derive. The resulting plasticity rule is defined locally so that global homeostasis near the critical state is achieved by local regulation of individual synapses. △ Less

Submitted 25 November, 2008; v1 submitted 22 August, 2008; originally announced August 2008.

Comments: 8 pages, 5 figures, after review NIPS'08

Novel Bounds on Marginal Probabilities

Authors: Joris M. Mooij, Hilbert J. Kappen

Abstract: We derive two related novel bounds on single-variable marginal probability distributions in factor graphs with discrete variables. The first method propagates bounds over a subtree of the factor graph rooted in the variable, and the second method propagates bounds over the self-avoiding walk tree starting at the variable. By construction, both methods not only bound the exact marginal probabilit… ▽ More We derive two related novel bounds on single-variable marginal probability distributions in factor graphs with discrete variables. The first method propagates bounds over a subtree of the factor graph rooted in the variable, and the second method propagates bounds over the self-avoiding walk tree starting at the variable. By construction, both methods not only bound the exact marginal probability distribution of a variable, but also its approximate Belief Propagation marginal (``belief''). Thus, apart from providing a practical means to calculate bounds on marginals, our contribution also lies in an increased understanding of the error made by Belief Propagation. Empirically, we show that our bounds often outperform existing bounds in terms of accuracy and/or computation time. We also show that our bounds can yield nontrivial results for medical diagnosis inference problems. △ Less

Submitted 24 January, 2008; originally announced January 2008.

Comments: 33 pages. Submitted to Journal of Machine Learning Research

MSC Class: 65C50

Truncating the loop series expansion for Belief Propagation

Authors: Vicenc Gomez, J. M. Mooij, H. J. Kappen

Abstract: Recently, M. Chertkov and V.Y. Chernyak derived an exact expression for the partition sum (normalization constant) corresponding to a graphical model, which is an expansion around the Belief Propagation solution. By adding correction terms to the BP free energy, one for each "generalized loop" in the factor graph, the exact partition sum is obtained. However, the usually enormous number of gener… ▽ More Recently, M. Chertkov and V.Y. Chernyak derived an exact expression for the partition sum (normalization constant) corresponding to a graphical model, which is an expansion around the Belief Propagation solution. By adding correction terms to the BP free energy, one for each "generalized loop" in the factor graph, the exact partition sum is obtained. However, the usually enormous number of generalized loops generally prohibits summation over all correction terms. In this article we introduce Truncated Loop Series BP (TLSBP), a particular way of truncating the loop series of M. Chertkov and V.Y. Chernyak by considering generalized loops as compositions of simple loops. We analyze the performance of TLSBP in different scenarios, including the Ising model, regular random graphs and on Promedas, a large probabilistic medical diagnostic system. We show that TLSBP often improves upon the accuracy of the BP solution, at the expense of increased computation time. We also show that the performance of TLSBP strongly depends on the degree of interaction between the variables. For weak interactions, truncating the series leads to significant improvements, whereas for strong interactions it can be ineffective, even if a high number of terms is considered. △ Less

Submitted 25 July, 2007; v1 submitted 21 December, 2006; originally announced December 2006.

Comments: 31 pages, 12 figures, submitted to Journal of Machine Learning Research

Journal ref: The Journal of Machine Learning Research, 8(Sep):1987--2016, 2007

On Cavity Approximations for Graphical Models

Authors: T. Rizzo, B. Wemmenhove, H. J. Kappen

Abstract: We reformulate the Cavity Approximation (CA), a class of algorithms recently introduced for improving the Bethe approximation estimates of marginals in graphical models. In our new formulation, which allows for the treatment of multivalued variables, a further generalization to factor graphs with arbitrary order of interaction factors is explicitly carried out, and a message passing algorithm th… ▽ More We reformulate the Cavity Approximation (CA), a class of algorithms recently introduced for improving the Bethe approximation estimates of marginals in graphical models. In our new formulation, which allows for the treatment of multivalued variables, a further generalization to factor graphs with arbitrary order of interaction factors is explicitly carried out, and a message passing algorithm that implements the first order correction to the Bethe approximation is described. Furthermore we investigate an implementation of the CA for pairwise interactions. In all cases considered we could confirm that CA[k] with increasing $k$ provides a sequence of approximations of markedly increasing precision. Furthermore in some cases we could also confirm the general expectation that the approximation of order $k$, whose computational complexity is $O(N^{k+1})$ has an error that scales as $1/N^{k+1}$ with the size of the system. We discuss the relation between this approach and some recent developments in the field. △ Less

Submitted 16 January, 2007; v1 submitted 14 August, 2006; originally announced August 2006.

Comments: Extension to factor graphs and comments on related work added

Competition between synaptic depression and facilitation in attractor neural networks

Authors: J. J. Torres, J. M. Cortes, J. Marro, H. J. Kappen

Abstract: We study the effect of competition between short-term synaptic depression and facilitation on the dynamical properties of attractor neural networks, using Monte Carlo simulation and a mean field analysis. Depending on the balance between depression, facilitation and the noise, the network displays different behaviours, including associative memory and switching of the activity between different… ▽ More We study the effect of competition between short-term synaptic depression and facilitation on the dynamical properties of attractor neural networks, using Monte Carlo simulation and a mean field analysis. Depending on the balance between depression, facilitation and the noise, the network displays different behaviours, including associative memory and switching of the activity between different attractors. We conclude that synaptic facilitation enhances the attractor instability in a way that (i) intensifies the system adaptability to external stimuli, which is in agreement with experiments, and (ii) favours the retrieval of information with less error during short--time intervals. △ Less

Submitted 16 April, 2006; originally announced April 2006.

Comments: 14 pages, 7 figures

Algorithms for identification and categorization

Authors: J. M. Cortes, P. L. Garrido, H. J. Kappen, J. Marro, C. Morillas, D. Navidad, J. J. Torres

Abstract: The main features of a family of efficient algorithms for recognition and classification of complex patterns are briefly reviewed. They are inspired in the observation that fast synaptic noise is essential for some of the processing of information in the brain. The main features of a family of efficient algorithms for recognition and classification of complex patterns are briefly reviewed. They are inspired in the observation that fast synaptic noise is essential for some of the processing of information in the brain. △ Less

Submitted 16 April, 2006; originally announced April 2006.

Comments: 6 pages, 5 figures

Journal ref: AIP Conference Proceedings 779: 178-184, 2005

Effects of fast presynaptic noise in attractor neural networks

Authors: J. M. Cortes, J. J. Torres, J. Marro, P. L. Garrido, H. J. Kappen

Abstract: We study both analytically and numerically the effect of presynaptic noise on the transmission of information in attractor neural networks. The noise occurs on a very short-time scale compared to that for the neuron dynamics and it produces short-time synaptic depression. This is inspired in recent neurobiological findings that show that synaptic strength may either increase or decrease on a sho… ▽ More We study both analytically and numerically the effect of presynaptic noise on the transmission of information in attractor neural networks. The noise occurs on a very short-time scale compared to that for the neuron dynamics and it produces short-time synaptic depression. This is inspired in recent neurobiological findings that show that synaptic strength may either increase or decrease on a short-time scale depending on presynaptic activity. We thus describe a mechanism by which fast presynaptic noise enhances the neural network sensitivity to an external stimulus. The reason for this is that, in general, the presynaptic noise induces nonequilibrium behavior and, consequently, the space of fixed points is qualitatively modified in such a way that the system can easily scape from the attractor. As a result, the model shows, in addition to pattern recognition, class identification and categorization, which may be relevant to the understanding of some of the brain complex tasks. △ Less

Submitted 13 August, 2005; originally announced August 2005.

Comments: 12 pages, 6 figures. To appear in Neural Computation, 2005