-
The Role of Learning Algorithms in Collective Action
Authors:
Omri Ben-Dov,
Jake Fawkes,
Samira Samadi,
Amartya Sanyal
Abstract:
Collective action in machine learning is the study of the control that a coordinated group can have over machine learning algorithms. While previous research has concentrated on assessing the impact of collectives against Bayes (sub-)optimal classifiers, this perspective is limited in that it does not account for the choice of learning algorithm. Since classifiers seldom behave like Bayes classifi…
▽ More
Collective action in machine learning is the study of the control that a coordinated group can have over machine learning algorithms. While previous research has concentrated on assessing the impact of collectives against Bayes (sub-)optimal classifiers, this perspective is limited in that it does not account for the choice of learning algorithm. Since classifiers seldom behave like Bayes classifiers and are influenced by the choice of learning algorithms along with their inherent biases, in this work we initiate the study of how the choice of the learning algorithm plays a role in the success of a collective in practical settings. Specifically, we focus on distributionally robust optimization (DRO), popular for improving a worst group error, and on the ubiquitous stochastic gradient descent (SGD), due to its inductive bias for "simpler" functions. Our empirical results, supported by a theoretical foundation, show that the effective size and success of the collective are highly dependent on properties of the learning algorithm. This highlights the necessity of taking the learning algorithm into account when studying the impact of collective action in machine learning.
△ Less
Submitted 4 June, 2024; v1 submitted 10 May, 2024;
originally announced May 2024.
-
Adversarial Likelihood Estimation With One-Way Flows
Authors:
Omri Ben-Dov,
Pravir Singh Gupta,
Victoria Abrevaya,
Michael J. Black,
Partha Ghosh
Abstract:
Generative Adversarial Networks (GANs) can produce high-quality samples, but do not provide an estimate of the probability density around the samples. However, it has been noted that maximizing the log-likelihood within an energy-based setting can lead to an adversarial framework where the discriminator provides unnormalized density (often called energy). We further develop this perspective, incor…
▽ More
Generative Adversarial Networks (GANs) can produce high-quality samples, but do not provide an estimate of the probability density around the samples. However, it has been noted that maximizing the log-likelihood within an energy-based setting can lead to an adversarial framework where the discriminator provides unnormalized density (often called energy). We further develop this perspective, incorporate importance sampling, and show that 1) Wasserstein GAN performs a biased estimate of the partition function, and we propose instead to use an unbiased estimator; and 2) when optimizing for likelihood, one must maximize generator entropy. This is hypothesized to provide a better mode coverage. Different from previous works, we explicitly compute the density of the generated samples. This is the key enabler to designing an unbiased estimator of the partition function and computation of the generator entropy term. The generator density is obtained via a new type of flow network, called one-way flow network, that is less constrained in terms of architecture, as it does not require a tractable inverse function. Our experimental results show that our method converges faster, produces comparable sample quality to GANs with similar architecture, successfully avoids over-fitting to commonly used datasets and produces smooth low-dimensional latent representations of the training data.
△ Less
Submitted 2 October, 2023; v1 submitted 19 July, 2023;
originally announced July 2023.
-
Solving the thoracic inverse problem in the fruit fly
Authors:
Arion Pons,
Illy Perl,
Omri Ben-Dov,
Roni Maya,
Tsevi Beatus
Abstract:
In many insect species, the thoracic structure plays a crucial role in enabling flight. In the dipteran indirect flight mechanism, the thorax acts as a transmission link between the flight muscles and the wings, and it is often thought to act as an elastic modulator: improving flight motor efficiency thorough linear or nonlinear resonance. But as peering closely into the drivetrain of tiny insects…
▽ More
In many insect species, the thoracic structure plays a crucial role in enabling flight. In the dipteran indirect flight mechanism, the thorax acts as a transmission link between the flight muscles and the wings, and it is often thought to act as an elastic modulator: improving flight motor efficiency thorough linear or nonlinear resonance. But as peering closely into the drivetrain of tiny insects is experimentally difficult, the nature of this elastic modulation, and any associated resonant effects, are unclear. Here, we present a new inverse-problem methodology to surmount this difficulty. In a data synthesis process, we integrate experimentally-observable aerodynamic and musculoskeletal data for the fruit fly D. melanogaster, and identify several surprising properties of the fly's thorax. We find that fruit flies have an energetic need for flight motor resonance: energy savings due to flight motor elasticity can be up be to 30%. However, the elasticity of the flight muscles themselves is sufficient - and sometimes more than sufficient - to ensure flight motor resonance. In fruit flies, the role of the thorax as an elastic modulator is likely to be insignificant. We discover also a fundamental link between the fruit fly wingbeat kinematics and musculature dynamics: wingbeat kinematics show key adaptions, including in wing elevation angle, that ensure that wingbeat load requirements match musculature load output capability. Together, these newly-identified properties lead to novel conceptual model of the fruit fly's flight motor: a strongly-nonlinear structure, resonant due to muscular elasticity, and intensely concerned with ensuring that the primary flight muscles are operating efficiently. Our inverse-problem methodology sheds new light on the complex behavior of these tiny flight motors, and provides avenues for further studies in a range of other insects.
△ Less
Submitted 27 September, 2022;
originally announced September 2022.
-
LED: Latent Variable-based Estimation of Density
Authors:
Omri Ben-Dov,
Pravir Singh Gupta,
Victoria Fernandez Abrevaya,
Michael J. Black,
Partha Ghosh
Abstract:
Modern generative models are roughly divided into two main categories: (1) models that can produce high-quality random samples, but cannot estimate the exact density of new data points and (2) those that provide exact density estimation, at the expense of sample quality and compactness of the latent space. In this work we propose LED, a new generative model closely related to GANs, that allows not…
▽ More
Modern generative models are roughly divided into two main categories: (1) models that can produce high-quality random samples, but cannot estimate the exact density of new data points and (2) those that provide exact density estimation, at the expense of sample quality and compactness of the latent space. In this work we propose LED, a new generative model closely related to GANs, that allows not only efficient sampling but also efficient density estimation. By maximizing log-likelihood on the output of the discriminator, we arrive at an alternative adversarial optimization objective that encourages generated data diversity. This formulation provides insights into the relationships between several popular generative models. Additionally, we construct a flow-based generator that can compute exact probabilities for generated samples, while allowing low-dimensional latent variables as input. Our experimental results, on various datasets, show that our density estimator produces accurate estimates, while retaining good quality in the generated samples.
△ Less
Submitted 23 June, 2022;
originally announced June 2022.
