Showing 1–12 of 12 results for author: Shaul, N

Corrector Sampling in Language Models

Authors: Itai Gat, Neta Shaul, Uriel Singer, Yaron Lipman

Abstract: Autoregressive language models accumulate errors due to their fixed, irrevocable left-to-right token generation. To address this, we propose a new sampling method called Resample-Previous-Tokens (RPT). RPT mitigates error accumulation by iteratively revisiting and potentially replacing tokens in a window of previously generated text. This method can be integrated into existing autoregressive model… ▽ More Autoregressive language models accumulate errors due to their fixed, irrevocable left-to-right token generation. To address this, we propose a new sampling method called Resample-Previous-Tokens (RPT). RPT mitigates error accumulation by iteratively revisiting and potentially replacing tokens in a window of previously generated text. This method can be integrated into existing autoregressive models, preserving their next-token-prediction quality and speed. Fine-tuning a pretrained 8B parameter model with RPT for only 100B resulted in ~10% relative improvements on reasoning and coding benchmarks compared to the standard sampling. △ Less

Submitted 6 June, 2025; originally announced June 2025.

Flow Matching Guide and Code

Authors: Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, Itai Gat

Abstract: Flow Matching (FM) is a recent framework for generative modeling that has achieved state-of-the-art performance across various domains, including image, video, audio, speech, and biological structures. This guide offers a comprehensive and self-contained review of FM, covering its mathematical foundations, design choices, and extensions. By also providing a PyTorch package featuring relevant examp… ▽ More Flow Matching (FM) is a recent framework for generative modeling that has achieved state-of-the-art performance across various domains, including image, video, audio, speech, and biological structures. This guide offers a comprehensive and self-contained review of FM, covering its mathematical foundations, design choices, and extensions. By also providing a PyTorch package featuring relevant examples (e.g., image and text generation), this work aims to serve as a resource for both novice and experienced researchers interested in understanding, applying and further developing FM. △ Less

Submitted 9 December, 2024; originally announced December 2024.

Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective

Authors: Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, Ricky T. Q. Chen

Abstract: The design space of discrete-space diffusion or flow generative models are significantly less well-understood than their continuous-space counterparts, with many works focusing only on a simple masked construction. In this work, we aim to take a holistic approach to the construction of discrete generative models based on continuous-time Markov chains, and for the first time, allow the use of arbit… ▽ More The design space of discrete-space diffusion or flow generative models are significantly less well-understood than their continuous-space counterparts, with many works focusing only on a simple masked construction. In this work, we aim to take a holistic approach to the construction of discrete generative models based on continuous-time Markov chains, and for the first time, allow the use of arbitrary discrete probability paths, or colloquially, corruption processes. Through the lens of optimizing the symmetric kinetic energy, we propose velocity formulas that can be applied to any given probability path, completely decoupling the probability and velocity, and giving the user the freedom to specify any desirable probability path based on expert knowledge specific to the data domain. Furthermore, we find that a special construction of mixture probability paths optimizes the symmetric kinetic energy for the discrete case. We empirically validate the usefulness of this new design space across multiple modalities: text generation, inorganic material generation, and image generation. We find that we can outperform the mask construction even in text with kinetic-optimal mixture paths, while we can make use of domain-specific constructions of the probability path over the visual domain. △ Less

Submitted 4 December, 2024; originally announced December 2024.

Generator Matching: Generative modeling with arbitrary Markov processes

Authors: Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, Yaron Lipman

Abstract: We introduce Generator Matching, a modality-agnostic framework for generative modeling using arbitrary Markov processes. Generators characterize the infinitesimal evolution of a Markov process, which we leverage for generative modeling in a similar vein to flow matching: we construct conditional generators which generate single data points, then learn to approximate the marginal generator which ge… ▽ More We introduce Generator Matching, a modality-agnostic framework for generative modeling using arbitrary Markov processes. Generators characterize the infinitesimal evolution of a Markov process, which we leverage for generative modeling in a similar vein to flow matching: we construct conditional generators which generate single data points, then learn to approximate the marginal generator which generates the full data distribution. We show that Generator Matching unifies various generative modeling methods, including diffusion models, flow matching and discrete diffusion models. Furthermore, it expands the design space to new and unexplored Markov processes such as jump processes. Finally, Generator Matching enables the construction of superpositions of Markov generative models and enables the construction of multimodal models in a rigorous manner. We empirically validate our method on image and multimodal generation, e.g. showing that superposition with a jump process improves performance. △ Less

Submitted 26 February, 2025; v1 submitted 27 October, 2024; originally announced October 2024.

Discrete Flow Matching

Authors: Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman

Abstract: Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers severa… ▽ More Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions:(i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($ε$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models. △ Less

Submitted 5 November, 2024; v1 submitted 22 July, 2024; originally announced July 2024.

Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models

Authors: Neta Shaul, Uriel Singer, Ricky T. Q. Chen, Matthew Le, Ali Thabet, Albert Pumarola, Yaron Lipman

Abstract: This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillatio… ▽ More This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillation, BNS solvers benefit from a tiny parameter space ($<$200 parameters), fast optimization (two orders of magnitude faster), maintain diversity of samples, and in contrast to previous solver distillation approaches nearly close the gap from standard distillation methods such as Progressive Distillation in the low-medium NFE regime. For example, BNS solver achieves 45 PSNR / 1.76 FID using 16 NFE in class-conditional ImageNet-64. We experimented with BNS solvers for conditional image generation, text-to-image generation, and text-2-audio generation showing significant improvement in sample approximation (PSNR) in all. △ Less

Submitted 2 March, 2024; originally announced March 2024.

Guided Flows for Generative Modeling and Decision Making

Authors: Qinqing Zheng, Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, Ricky T. Q. Chen

Abstract: Classifier-free guidance is a key component for enhancing the performance of conditional generative models across diverse tasks. While it has previously demonstrated remarkable improvements for the sample quality, it has only been exclusively employed for diffusion models. In this paper, we integrate classifier-free guidance into Flow Matching (FM) models, an alternative simulation-free approach t… ▽ More Classifier-free guidance is a key component for enhancing the performance of conditional generative models across diverse tasks. While it has previously demonstrated remarkable improvements for the sample quality, it has only been exclusively employed for diffusion models. In this paper, we integrate classifier-free guidance into Flow Matching (FM) models, an alternative simulation-free approach that trains Continuous Normalizing Flows (CNFs) based on regressing vector fields. We explore the usage of \emph{Guided Flows} for a variety of downstream applications. We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text-to-speech synthesis, boasting state-of-the-art performance. Notably, we are the first to apply flow models for plan generation in the offline reinforcement learning setting, showcasing a 10x speedup in computation compared to diffusion models while maintaining comparable performance. △ Less

Submitted 7 December, 2023; v1 submitted 22 November, 2023; originally announced November 2023.

Bespoke Solvers for Generative Flow Models

Authors: Neta Shaul, Juan Perez, Ricky T. Q. Chen, Ali Thabet, Albert Pumarola, Yaron Lipman

Abstract: Diffusion or flow-based models are powerful generative paradigms that are notoriously hard to sample as samples are defined as solutions to high-dimensional Ordinary or Stochastic Differential Equations (ODEs/SDEs) which require a large Number of Function Evaluations (NFE) to approximate well. Existing methods to alleviate the costly sampling process include model distillation and designing dedica… ▽ More Diffusion or flow-based models are powerful generative paradigms that are notoriously hard to sample as samples are defined as solutions to high-dimensional Ordinary or Stochastic Differential Equations (ODEs/SDEs) which require a large Number of Function Evaluations (NFE) to approximate well. Existing methods to alleviate the costly sampling process include model distillation and designing dedicated ODE solvers. However, distillation is costly to train and sometimes can deteriorate quality, while dedicated solvers still require relatively large NFE to produce high quality samples. In this paper we introduce "Bespoke solvers", a novel framework for constructing custom ODE solvers tailored to the ODE of a given pre-trained flow model. Our approach optimizes an order consistent and parameter-efficient solver (e.g., with 80 learnable parameters), is trained for roughly 1% of the GPU time required for training the pre-trained model, and significantly improves approximation and generation quality compared to dedicated solvers. For example, a Bespoke solver for a CIFAR10 model produces samples with Fréchet Inception Distance (FID) of 2.73 with 10 NFE, and gets to 1% of the Ground Truth (GT) FID (2.59) for this model with only 20 NFE. On the more challenging ImageNet-64$\times$64, Bespoke samples at 2.2 FID with 10 NFE, and gets within 2% of GT FID (1.71) with 20 NFE. △ Less

Submitted 29 October, 2023; originally announced October 2023.

On Kinetic Optimal Probability Paths for Generative Models

Authors: Neta Shaul, Ricky T. Q. Chen, Maximilian Nickel, Matt Le, Yaron Lipman

Abstract: Recent successful generative models are trained by fitting a neural network to an a-priori defined tractable probability density path taking noise to training examples. In this paper we investigate the space of Gaussian probability paths, which includes diffusion paths as an instance, and look for an optimal member in some useful sense. In particular, minimizing the Kinetic Energy (KE) of a path i… ▽ More Recent successful generative models are trained by fitting a neural network to an a-priori defined tractable probability density path taking noise to training examples. In this paper we investigate the space of Gaussian probability paths, which includes diffusion paths as an instance, and look for an optimal member in some useful sense. In particular, minimizing the Kinetic Energy (KE) of a path is known to make particles' trajectories simple, hence easier to sample, and empirically improve performance in terms of likelihood of unseen data and sample generation quality. We investigate Kinetic Optimal (KO) Gaussian paths and offer the following observations: (i) We show the KE takes a simplified form on the space of Gaussian paths, where the data is incorporated only through a single, one dimensional scalar function, called the \emph{data separation function}. (ii) We characterize the KO solutions with a one dimensional ODE. (iii) We approximate data-dependent KO paths by approximating the data separation function and minimizing the KE. (iv) We prove that the data separation function converges to $1$ in the general case of arbitrary normalized dataset consisting of $n$ samples in $d$ dimension as $n/\sqrt{d}\rightarrow 0$. A consequence of this result is that the Conditional Optimal Transport (Cond-OT) path becomes \emph{kinetic optimal} as $n/\sqrt{d}\rightarrow 0$. We further support this theory with empirical experiments on ImageNet. △ Less

Submitted 11 June, 2023; originally announced June 2023.

Improved Simulation of the Mass Charging for ASTROD I

Authors: Gang Bao, Wei-Tou Ni, D. N. A. Shaul, H. M. Araujo, Lei Liu, T. J. Sumner

Abstract: The electrostatic charging of the test mass in ASTROD I (Astrodynamical Space Test of Relativity using Optical Devices I) mission can affect the quality of the science data as a result of spurious Coulomb and Lorentz forces. To estimate the size of the resultant disturbances, credible predictions of charging rates and the charging noise are required. Using the GEANT4 software toolkit, we present… ▽ More The electrostatic charging of the test mass in ASTROD I (Astrodynamical Space Test of Relativity using Optical Devices I) mission can affect the quality of the science data as a result of spurious Coulomb and Lorentz forces. To estimate the size of the resultant disturbances, credible predictions of charging rates and the charging noise are required. Using the GEANT4 software toolkit, we present a detailed Monte Carlo simulation of the ASTROD I test mass charging due to exposure of the spacecraft to galactic cosmic-ray (GCR) protons and alpha particles (3He, 4He) in the space environment. A positive charging rate of 33.3 e+/s at solar minimum is obtained. This figure reduces by 50% at solar maximum. Based on this charging rate and factoring in the contribution of minor cosmic-ray components, we calculate the acceleration noise and stiffness associated with charging. We conclude that the acceleration noise arising from Coulomb and Lorentz effects are well below the ASTROD I acceleration noise limit at 0.1 mHz both at solar minimum and maximum. The coherent Fourier components due to charging are investigated, it needs to be studied carefully in order to ensure that these do not compromise the quality of science data in the ASTROD I mission. △ Less

Submitted 13 July, 2007; originally announced July 2007.

Comments: 20 pages, 14 figures, submitted to International Journal of Modern Physics D

Journal ref: INTERNATIONAL JOURNAL OF MODERN PHYSICS D, Vol.17, No.7, 965 (July 2008)

Simulation of ASTROD I test mass charging due to solar energetic particles

Authors: Lei Liu, Gang Bao, Wei-Tou Ni, D N A Shaul

Abstract: As ASTROD I travels through space, its test mass will accrue charge due to galactic cosmic-rays and solar energetic particles incident on the spacecraft. This test mass charge will result in Coulomb forces between the test mass and the surrounding electrodes. In earlier work using the GEANT4 toolkit, we predicted a net charging rate of nearly 9.0 +e/s from cosmic-ray protons between 0.1 and 1000… ▽ More As ASTROD I travels through space, its test mass will accrue charge due to galactic cosmic-rays and solar energetic particles incident on the spacecraft. This test mass charge will result in Coulomb forces between the test mass and the surrounding electrodes. In earlier work using the GEANT4 toolkit, we predicted a net charging rate of nearly 9.0 +e/s from cosmic-ray protons between 0.1 and 1000 GeV at solar maximum, and rising to 26.5 +e/s at solar minimum. Here we use GEANT4 to simulate the charging process due to solar energetic particle events and to estimate the magnitude of acceleration noise due to this charging. The predicted charging rates range from 2840 to 64300 +e/s, at peak intensity, for the 4 largest SEP events in September and October 1989. For the 2 larger events, the acceleration disturbances due to charging exceeds the ASTROD I acceleration noise budget requirement. Continuous discharge should be considered for suppressing this charging noise. The acceleration noise during the 2 small events is well below the design target, although during these events, the net charging rate will be dominated by these solar fluxes. △ Less

Submitted 26 April, 2007; originally announced April 2007.

Comments: 10 pages,8 figures, COSPAR2006 H0.1-1, submitted

ASTROD I Charging Simulation and Disturbances

Authors: Gang Bao, D N A Shaul, H M Araujo, Wei-Tou Ni, T J Sumner, Lei Liu

Abstract: ASTROD I is planned as a single spacecraft mission. It will use interferometric and pulse ranging techniques between the spacecraft and ground stations, to make high precision measurements of the parameters that describe the solar system, and to test relativistic gravity with improved accuracy. At the heart of the spacecraft is a test mass, which the spacecraft will follow using a drag-free cont… ▽ More ASTROD I is planned as a single spacecraft mission. It will use interferometric and pulse ranging techniques between the spacecraft and ground stations, to make high precision measurements of the parameters that describe the solar system, and to test relativistic gravity with improved accuracy. At the heart of the spacecraft is a test mass, which the spacecraft will follow using a drag-free control system. The mission critically depends on maintaining the geodesic motion of the test mass. Charging of the test mass due to cosmic rays and solar particles will disturb its geodesic motion. We have modelled the charging process using the GEANT4 toolkit and a simplified, geometrical model and estimate that the ASTROD I test mass will charge positively, at a rate of 24 +/-7 e+/s, due to cosmic ray protons and alpha particles (3He and 4He) at solar minimum. We have used the results of this simulation to estimate the magnitude of disturbances associated with test mass charging, for the worst-case scenario, taking into account uncertainties in the model and potential charging contributions from minor cosmic-ray components. △ Less

Submitted 24 April, 2007; originally announced April 2007.

Comments: 17 pages, 11 figures, submitted to General Relativity and Gravitation