-
Generative diffusion models from a PDE perspective
Authors:
Fei Cao,
Kimball Johnston,
Thomas Laurent,
Justin Le,
Sébastien Motsch
Abstract:
Diffusion models have become the de facto framework for generating new datasets. The core of these models lies in the ability to reverse a diffusion process in time. The goal of this manuscript is to explain, from a PDE perspective, how this method works and how to derive the PDE governing the reverse dynamics as well as to study its solution analytically. By linking forward and reverse dynamics,…
▽ More
Diffusion models have become the de facto framework for generating new datasets. The core of these models lies in the ability to reverse a diffusion process in time. The goal of this manuscript is to explain, from a PDE perspective, how this method works and how to derive the PDE governing the reverse dynamics as well as to study its solution analytically. By linking forward and reverse dynamics, we show that the reverse process's distribution has its support contained within the original distribution. Consequently, diffusion methods, in their analytical formulation, do not inherently regularize the original distribution, and thus, there is no generalization principle. This raises a question: where does generalization arise, given that in practice it does occur? Moreover, we derive an explicit solution to the reverse process's SDE under the assumption that the starting point of the forward process is fixed. This provides a new derivation that links two popular approaches to generative diffusion models: stable diffusion (discrete dynamics) and the score-based approach (continuous dynamics). Finally, we explore the case where the original distribution consists of a finite set of data points. In this scenario, the reverse dynamics are explicit (i.e., the loss function has a clear minimizer), and solving the dynamics fails to generate new samples: the dynamics converge to the original samples. In a sense, solving the minimization problem exactly is "too good for its own good" (i.e., an overfitting regime).
△ Less
Submitted 28 January, 2025;
originally announced January 2025.
-
Sticky dispersion on the complete graph: a kinetic approach
Authors:
Fei Cao,
Sebastien Motsch
Abstract:
We study a variant of the dispersion process on the complete graph introduced in the recent work [17] under the mean-field framework. We adopt a kinetic perspective (as opposed to the probabilistic approach taken in [17] and many other related works) thanks to the reinterpretation of the model in terms of a novel econophysics model. Various analytical and quantitative results regarding the large t…
▽ More
We study a variant of the dispersion process on the complete graph introduced in the recent work [17] under the mean-field framework. We adopt a kinetic perspective (as opposed to the probabilistic approach taken in [17] and many other related works) thanks to the reinterpretation of the model in terms of a novel econophysics model. Various analytical and quantitative results regarding the large time behaviour of the mean-field dynamics are obtained and supporting numerical illustrations are provided.
△ Less
Submitted 12 April, 2024;
originally announced April 2024.
-
Uncovering a two-phase dynamics from a dollar exchange model with bank and debt
Authors:
Fei Cao,
Sébastien Motsch
Abstract:
We investigate the unbiased model for money exchanges with collective debt limit: agents give at random time a dollar to one another as long as they have at least one dollar or they can borrow a dollar from a central bank if the bank is not empty. Surprisingly, this dynamics eventually leads to an asymmetric Laplace distribution of wealth (conjectured in [22] and shown formally in a recent work [1…
▽ More
We investigate the unbiased model for money exchanges with collective debt limit: agents give at random time a dollar to one another as long as they have at least one dollar or they can borrow a dollar from a central bank if the bank is not empty. Surprisingly, this dynamics eventually leads to an asymmetric Laplace distribution of wealth (conjectured in [22] and shown formally in a recent work [18]). In this manuscript, we carry out a formal mean-field limit as the number of agents goes to infinity where we uncover a two-phase (ODE) dynamics. Convergence towards the unique equilibrium (two-sided geometric) distribution in the large time limit is also shown and the role played by the bank and debt (in terms of Gini index or wealth inequality) will be explored numerically as well.
△ Less
Submitted 23 August, 2022;
originally announced August 2022.
-
Derivation of wealth distributions from biased exchange of money
Authors:
Fei Cao,
Sebastien Motsch
Abstract:
In the manuscript, we are interested in using kinetic theory to better understand the time evolution of wealth distribution and their large scale behavior such as the evolution of inequality (e.g. Gini index). We investigate three type of dynamics denoted unbiased, poor-biased and rich-biased dynamics. At the particle level, one agent is picked randomly based on its wealth and one of its dollar is…
▽ More
In the manuscript, we are interested in using kinetic theory to better understand the time evolution of wealth distribution and their large scale behavior such as the evolution of inequality (e.g. Gini index). We investigate three type of dynamics denoted unbiased, poor-biased and rich-biased dynamics. At the particle level, one agent is picked randomly based on its wealth and one of its dollar is redistributed among the population. Proving the so-called propagation of chaos, we identify the limit of each dynamics as the number of individual approaches infinity using both coupling techniques [48] and martingale-based approach [36]. Equipped with the limit equation, we identify and prove the convergence to specific equilibrium for both the unbiased and poor-biased dynamics. In the rich-biased dynamics however, we observe a more complex behavior where a dispersive wave emerges. Although the dispersive wave is vanishing in time, its also accumulates all the wealth leading to a Gini approaching 1 (its maximum value). We characterize numerically the behavior of dispersive wave but further analytic investigation is needed to derive such dispersive wave directly from the dynamics.
△ Less
Submitted 15 May, 2021;
originally announced May 2021.
-
Entropy dissipation and propagation of chaos for the uniform reshuffling model
Authors:
Fei Cao,
Pierre-Emmanuel Jabin,
Sebastien Motsch
Abstract:
We investigate the uniform reshuffling model for money exchanges: two agents picked uniformly at random redistribute their dollars between them. This stochastic dynamics is of mean-field type and eventually leads to a exponential distribution of wealth. To better understand this dynamics, we investigate its limit as the number of agents goes to infinity. We prove rigorously the so-called propagati…
▽ More
We investigate the uniform reshuffling model for money exchanges: two agents picked uniformly at random redistribute their dollars between them. This stochastic dynamics is of mean-field type and eventually leads to a exponential distribution of wealth. To better understand this dynamics, we investigate its limit as the number of agents goes to infinity. We prove rigorously the so-called propagation of chaos which links the stochastic dynamics to a (limiting) nonlinear partial differential equation (PDE). This deterministic description, which is well-known in the literature, has a flavor of the classical Boltzmann equation arising from statistical mechanics of dilute gases. We prove its convergence toward its exponential equilibrium distribution in the sense of relative entropy.
△ Less
Submitted 2 April, 2021;
originally announced April 2021.
-
Bounded confidence dynamics and graph control: enforcing consensus
Authors:
Dylan Weber,
Sebastien Motsch,
GuanLin Li
Abstract:
A generic feature of bounded confidence type models is the formation of clusters of agents. We propose and study a variant of bounded confidence dynamics with the goal of inducing unconditional convergence to a consensus. The defining feature of these dynamics, which we name the No one left behind dynamics, is the introduction of a local control on the agents which preserves the connectivity of th…
▽ More
A generic feature of bounded confidence type models is the formation of clusters of agents. We propose and study a variant of bounded confidence dynamics with the goal of inducing unconditional convergence to a consensus. The defining feature of these dynamics, which we name the No one left behind dynamics, is the introduction of a local control on the agents which preserves the connectivity of the interaction network. We rigorously demonstrate that these dynamics result in unconditional convergence to a consensus. The qualitative nature of our argument prevents us quantifying how fast a consensus emerges, however we present numerical evidence that sharp convergence rates would be challenging to obtain for such dynamics. Finally, we propose a relaxed version of the control. The dynamics that result maintain many of the qualitative features of the bounded confidence dynamics yet ultimately still converge to a consensus as the control still maintains connectivity of the interaction network.
△ Less
Submitted 18 June, 2020;
originally announced June 2020.
-
Stochastic versus deterministic consensus dynamics on graphs
Authors:
Dylan Weber,
Ryan Theisen,
Sebastien Motsch
Abstract:
We study two agent based models of opinion formation - one stochastic in nature and one deterministic. Both models are defined in terms of an underlying graph; we study how the structure of the graph affects the long time behavior of the models in all possible cases of graph topology. We are especially interested in the emergence of a consensus among the agents and provide a condition on the graph…
▽ More
We study two agent based models of opinion formation - one stochastic in nature and one deterministic. Both models are defined in terms of an underlying graph; we study how the structure of the graph affects the long time behavior of the models in all possible cases of graph topology. We are especially interested in the emergence of a consensus among the agents and provide a condition on the graph that is necessary and sufficient for convergence to a consensus in both models. This investigation reveals several contrasts between the models - notably the convergence rates - which are explored through analytical arguments and several numerical experiments.
△ Less
Submitted 30 January, 2019;
originally announced January 2019.
-
Kinetic equations and self-organized band formations
Authors:
Quentin Griette,
Sebastien Motsch
Abstract:
Self-organization is an ubiquitous phenomenon in nature which can be observed in a variety of different contexts and scales, with examples ranging from fish schools, swarms of birds or locusts, to flocks of bacteria. The observation of such global patterns can often be reproduced in models based on simple interactions between neighboring particles. In this paper we focus on two particular interact…
▽ More
Self-organization is an ubiquitous phenomenon in nature which can be observed in a variety of different contexts and scales, with examples ranging from fish schools, swarms of birds or locusts, to flocks of bacteria. The observation of such global patterns can often be reproduced in models based on simple interactions between neighboring particles. In this paper we focus on two particular interaction dynamics closely related to the one described in the seminal paper of Vicsek and collaborators. After reviewing the current state of the art in the subject, we study a numerical scheme for the kinetic equation associated with the Vicsek models which has the specificity of reproducing many physical properties of the continuous models, like the preservation of energy and positivity and the diminution of an entropy functional. We describe a stable pattern of bands emerging in the dynamics proposed by Degond-Frouvelle-Liu dynamics and give some insights about their formation.
△ Less
Submitted 14 January, 2019;
originally announced January 2019.
-
Damped Arrow-Hurwicz algorithm for sphere packing
Authors:
Pierre Degond,
Marina A. Ferreira,
Sébastien Motsch
Abstract:
We consider algorithms that, from an arbitrarily sampling of $N$ spheres (possibly overlapping), find a close packed configuration without overlapping. These problems can be formulated as minimization problems with non-convex constraints. For such packing problems, we observe that the classical iterative Arrow-Hurwicz algorithm does not converge. We derive a novel algorithm from a multi-step varia…
▽ More
We consider algorithms that, from an arbitrarily sampling of $N$ spheres (possibly overlapping), find a close packed configuration without overlapping. These problems can be formulated as minimization problems with non-convex constraints. For such packing problems, we observe that the classical iterative Arrow-Hurwicz algorithm does not converge. We derive a novel algorithm from a multi-step variant of the Arrow-Hurwicz scheme with damping. We compare this algorithm with classical algorithms belonging to the class of linearly constrained Lagrangian methods and show that it performs better. We provide an analysis of the convergence of these algorithms in the simple case of two spheres in one spatial dimension. Finally, we investigate the behaviour of our algorithm when the number of spheres is large.
△ Less
Submitted 18 May, 2016;
originally announced May 2016.
-
A new model for self-organized dynamics and its flocking behavior
Authors:
Sebastien Motsch,
Eitan Tadmor
Abstract:
We introduce a model for self-organized dynamics which, we argue, addresses several drawbacks of the celebrated Cucker-Smale (C-S) model. The proposed model does not only take into account the distance between agents, but instead, the influence between agents is scaled in term of their relative distance. Consequently, our model does not involve any explicit dependence on the number of agents; only…
▽ More
We introduce a model for self-organized dynamics which, we argue, addresses several drawbacks of the celebrated Cucker-Smale (C-S) model. The proposed model does not only take into account the distance between agents, but instead, the influence between agents is scaled in term of their relative distance. Consequently, our model does not involve any explicit dependence on the number of agents; only their geometry in phase space is taken into account. The use of relative distances destroys the symmetry property of the original C-S model, which was the key for the various recent studies of C-S flocking behavior. To this end, we introduce here a new framework to analyze the phenomenon of flocking for a rather general class of dynamical systems, which covers systems with non-symmetric influence matrices. In particular, we analyze the flocking behavior of the proposed model as well as other strongly asymmetric models with "leaders".
The methodology presented in this paper, based on the notion of active sets, carries over from the particle to kinetic and hydrodynamic descriptions. In particular, we discuss the hydrodynamic formulation of our proposed model, and prove its unconditional flocking for slowly decaying influence functions.
△ Less
Submitted 25 July, 2011; v1 submitted 27 February, 2011;
originally announced February 2011.
-
A macroscopic model for a system of swarming agents using curvature control
Authors:
Pierre Degond,
Sébastien Motsch
Abstract:
In this paper, we study the macroscopic limit of a new model of collective displacement. The model, called PTWA, is a combination of the Vicsek alignment model and the Persistent Turning Walker (PTW) model of motion by curvature control. The PTW model was designed to fit measured trajectories of individual fish. The PTWA model (Persistent Turning Walker with Alignment) describes the displacements…
▽ More
In this paper, we study the macroscopic limit of a new model of collective displacement. The model, called PTWA, is a combination of the Vicsek alignment model and the Persistent Turning Walker (PTW) model of motion by curvature control. The PTW model was designed to fit measured trajectories of individual fish. The PTWA model (Persistent Turning Walker with Alignment) describes the displacements of agents which modify their curvature in order to align with their neighbors. The derivation of its macroscopic limit uses the non-classical notion of generalized collisional invariant. The macroscopic limit of the PTWA model involves two physical quantities, the density and the mean velocity of individuals. It is a system of hyperbolic type but is non-conservative due to a geometric constraint on the velocity. This system has the same form as the macroscopic limit of the Vicsek model (the 'Vicsek hydrodynamics') but for the expression of the model coefficients. The numerical computations show that the numerical values of the coefficients are very close. The 'Vicsek Hydrodynamic model' appears in this way as a more generic macroscopic model of swarming behavior as originally anticipated.
△ Less
Submitted 26 October, 2010;
originally announced October 2010.
-
Numerical simulations of a non-conservative hyperbolic system with geometric constraints describing swarming behavior
Authors:
Sebastien Motsch,
Laurent Navoret
Abstract:
The Vicsek model is a very popular individual based model which describes collective behavior among animal societies. A macroscopic version of the Vicsek model has been derived from a large scale limit of this individual based model \cite{degond2007ml}. In this work, we want to numerically validate this Macroscopic Vicsek model (MV). To this aim, we compare the simulations of the macroscopic and…
▽ More
The Vicsek model is a very popular individual based model which describes collective behavior among animal societies. A macroscopic version of the Vicsek model has been derived from a large scale limit of this individual based model \cite{degond2007ml}. In this work, we want to numerically validate this Macroscopic Vicsek model (MV). To this aim, we compare the simulations of the macroscopic and microscopic models one with each other. The MV model is a non-conservative hyperbolic equation with a geometric constraint. Due to the lack of theory for this kind of equations, we derive several equivalents for this system leading to specific numerical schemes. The numerical simulations reveal that the microscopic and macroscopic models are in good agreement provided that we choose one of the proposed formulations based on a relaxation of the geometric constraint. This confirms the relevance of the macroscopic equation but it also calls for a better theoretical understanding of this type of equations.
△ Less
Submitted 15 October, 2009;
originally announced October 2009.
-
Asymptotic analysis and diffusion limit of the Persistent Turning Walker Model
Authors:
Patrick Cattiaux,
Djalil Chafai,
Sébastien Motsch
Abstract:
The Persistent Turning Walker Model (PTWM) was introduced by Gautrais et al in Mathematical Biology for the modelling of fish motion. It involves a nonlinear pathwise functional of a non-elliptic hypo-elliptic diffusion. This diffusion solves a kinetic Fokker-Planck equation based on an Ornstein-Uhlenbeck Gaussian process. The long time "diffusive" behavior of this model was recently studied by…
▽ More
The Persistent Turning Walker Model (PTWM) was introduced by Gautrais et al in Mathematical Biology for the modelling of fish motion. It involves a nonlinear pathwise functional of a non-elliptic hypo-elliptic diffusion. This diffusion solves a kinetic Fokker-Planck equation based on an Ornstein-Uhlenbeck Gaussian process. The long time "diffusive" behavior of this model was recently studied by Degond & Motsch using partial differential equations techniques. This model is however intrinsically probabilistic. In the present paper, we show how the long time diffusive behavior of this model can be essentially recovered and extended by using appropriate tools from stochastic analysis. The approach can be adapted to many other kinetic "probabilistic" models.
△ Less
Submitted 28 September, 2009; v1 submitted 4 November, 2008;
originally announced November 2008.
