Skip to main content

Showing 1–23 of 23 results for author: Vandermeulen, R A

.
  1. arXiv:2411.15095  [pdf, other

    stat.ML cs.CV cs.LG math.ST

    Dimension-independent rates for structured neural density estimation

    Authors: Robert A. Vandermeulen, Wai Ming Tai, Bryon Aragam

    Abstract: We show that deep neural networks achieve dimension-independent rates of convergence for learning structured densities such as those arising in image, audio, video, and text applications. More precisely, we demonstrate that neural networks with a simple $L^2$-minimizing loss achieve a rate of $n^{-1/(4+r)}$ in nonparametric density estimation when the underlying density is Markov to a graph whose… ▽ More

    Submitted 22 November, 2024; originally announced November 2024.

    MSC Class: 62G05; 62G07; 62A09; 62M05; 62M40; 60J10; 60J20 ACM Class: G.3; I.5.1; I.4.10; I.4.m

  2. arXiv:2410.07685  [pdf, other

    stat.ML cs.CV cs.LG math.ST

    Breaking the curse of dimensionality in structured density estimation

    Authors: Robert A. Vandermeulen, Wai Ming Tai, Bryon Aragam

    Abstract: We consider the problem of estimating a structured multivariate density, subject to Markov conditions implied by an undirected graph. In the worst case, without Markovian assumptions, this problem suffers from the curse of dimensionality. Our main result shows how the curse of dimensionality can be avoided or greatly alleviated under the Markov property, and applies to arbitrary graphs. While exis… ▽ More

    Submitted 10 October, 2024; originally announced October 2024.

    Comments: Work accepted to NeurIPS 2024

    MSC Class: 62G05; 62G07; 62A09; 62M05; 62M40; 60J10; 60J20 ACM Class: G.3; I.5.1

  3. arXiv:2307.02245  [pdf, other

    cs.LG cs.CV cs.IT

    Set Learning for Accurate and Calibrated Models

    Authors: Lukas Muttenthaler, Robert A. Vandermeulen, Qiuyi Zhang, Thomas Unterthiner, Klaus-Robert Müller

    Abstract: Model overconfidence and poor calibration are common in machine learning and difficult to account for when applying standard empirical risk minimization. In this work, we propose a novel method to alleviate these problems that we call odd-$k$-out learning (OKO), which minimizes the cross-entropy error for sets rather than for single examples. This naturally allows the model to capture correlations… ▽ More

    Submitted 12 February, 2024; v1 submitted 5 July, 2023; originally announced July 2023.

    Comments: Published as a conference paper at ICLR 2024

  4. arXiv:2306.04507  [pdf, other

    cs.CV cs.LG

    Improving neural network representations using human similarity judgments

    Authors: Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A. Vandermeulen, Katherine Hermann, Andrew K. Lampinen, Simon Kornblith

    Abstract: Deep neural networks have reached human-level performance on many computer vision tasks. However, the objectives used to train these networks enforce only that similar images are embedded at similar locations in the representation space, and do not directly constrain the global structure of the resulting space. Here, we explore the impact of supervising this global structure by linearly aligning i… ▽ More

    Submitted 26 September, 2023; v1 submitted 7 June, 2023; originally announced June 2023.

    Comments: Published as a conference paper at NeurIPS 2023

  5. arXiv:2302.04292  [pdf, ps, other

    math.ST stat.ML

    Sample Complexity Using Infinite Multiview Models

    Authors: Robert A. Vandermeulen

    Abstract: Recent works have demonstrated that the convergence rate of a nonparametric density estimator can be greatly improved by using a low-rank estimator when the target density is a convex combination of separable probability densities with Lipschitz continuous marginals, i.e. a multiview model. However, this assumption is very restrictive and it is not clear to what degree these findings can be extend… ▽ More

    Submitted 8 February, 2023; originally announced February 2023.

    MSC Class: 62G05; 62G07

  6. arXiv:2211.01201  [pdf, other

    cs.CV cs.AI cs.LG q-bio.NC

    Human alignment of neural network representations

    Authors: Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, Simon Kornblith

    Abstract: Today's computer vision models achieve human or near-human level performance across a wide variety of vision tasks. However, their architectures, data, and learning algorithms differ in numerous ways from those that give rise to human vision. In this paper, we investigate the factors that affect the alignment between the representations learned by neural networks and human mental representations i… ▽ More

    Submitted 16 February, 2025; v1 submitted 2 November, 2022; originally announced November 2022.

    Comments: Accepted for publication at ICLR 2023

  7. arXiv:2207.11164  [pdf, other

    math.ST cs.LG stat.ML

    Generalized Identifiability Bounds for Mixture Models with Grouped Samples

    Authors: Robert A. Vandermeulen, René Saitenmacher

    Abstract: Recent work has shown that finite mixture models with $m$ components are identifiable, while making no assumptions on the mixture components, so long as one has access to groups of samples of size $2m-1$ which are known to come from the same mixture component. In this work we generalize that result and show that, if every subset of $k$ mixture components of a mixture model are linearly independent… ▽ More

    Submitted 22 July, 2022; originally announced July 2022.

    MSC Class: 62G07; 62G05

  8. arXiv:2205.11474  [pdf, other

    cs.CV cs.LG stat.ML

    Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images

    Authors: Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Klaus-Robert Müller, Marius Kloft

    Abstract: Due to the intractability of characterizing everything that looks unlike the normal data, anomaly detection (AD) is traditionally treated as an unsupervised problem utilizing only normal samples. However, it has recently been found that unsupervised image AD can be drastically improved through the utilization of huge corpora of random images to represent anomalousness; a technique which is known a… ▽ More

    Submitted 14 November, 2022; v1 submitted 23 May, 2022; originally announced May 2022.

    Comments: 47 pages; extended experiments; published in Transactions on Machine Learning Research. arXiv admin note: substantial text overlap with arXiv:2006.00339

  9. arXiv:2205.00756  [pdf, other

    cs.LG stat.AP stat.ML

    VICE: Variational Interpretable Concept Embeddings

    Authors: Lukas Muttenthaler, Charles Y. Zheng, Patrick McClure, Robert A. Vandermeulen, Martin N. Hebart, Francisco Pereira

    Abstract: A central goal in the cognitive sciences is the development of numerical models for mental representations of object concepts. This paper introduces Variational Interpretable Concept Embeddings (VICE), an approximate Bayesian method for embedding object concepts in a vector space using data collected from humans in a triplet odd-one-out task. VICE uses variational inference to obtain sparse, non-n… ▽ More

    Submitted 6 October, 2022; v1 submitted 2 May, 2022; originally announced May 2022.

    Comments: Accepted at NeurIPS 2022

  10. arXiv:2204.00930  [pdf, ps, other

    math.ST stat.ML

    Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation

    Authors: Robert A. Vandermeulen, Antoine Ledent

    Abstract: The construction and theoretical analysis of the most popular universally consistent nonparametric density estimators hinge on one functional property: smoothness. In this paper we investigate the theoretical implications of incorporating a multi-view latent variable model, a type of low-rank model, into nonparametric density estimation. To do this we perform extensive analysis on histogram-style… ▽ More

    Submitted 2 April, 2022; originally announced April 2022.

    Comments: Accepted to NeurIPS 2021

  11. Learning Interpretable Concept Groups in CNNs

    Authors: Saurabh Varshneya, Antoine Ledent, Robert A. Vandermeulen, Yunwen Lei, Matthias Enders, Damian Borth, Marius Kloft

    Abstract: We propose a novel training methodology -- Concept Group Learning (CGL) -- that encourages training of interpretable CNN filters by partitioning filters in each layer into concept groups, each of which is trained to learn a single visual concept. We achieve this through a novel regularization strategy that forces filters in the same group to be active in similar image regions for a given layer. We… ▽ More

    Submitted 21 September, 2021; originally announced September 2021.

  12. arXiv:2010.02425  [pdf, ps, other

    math.ST cs.LG stat.ME stat.ML

    Improving Nonparametric Density Estimation with Tensor Decompositions

    Authors: Robert A. Vandermeulen

    Abstract: While nonparametric density estimators often perform well on low dimensional data, their performance can suffer when applied to higher dimensional data, owing presumably to the curse of dimensionality. One technique for avoiding this is to assume no dependence between features and that the data are sampled from a separable density. This allows one to estimate each marginal distribution independent… ▽ More

    Submitted 5 October, 2020; originally announced October 2020.

    Comments: 20 pages, 1 table

    MSC Class: 62G05; 62G07 ACM Class: G.3

  13. arXiv:2010.02310  [pdf, other

    cs.LG stat.ML

    Deep Anomaly Detection by Residual Adaptation

    Authors: Lucas Deecke, Lukas Ruff, Robert A. Vandermeulen, Hakan Bilen

    Abstract: Deep anomaly detection is a difficult task since, in high dimensions, it is hard to completely characterize a notion of "differentness" when given only examples of normality. In this paper we propose a novel approach to deep anomaly detection based on augmenting large pretrained networks with residual corrections that adjusts them to the task of anomaly detection. Our method gives rise to a highly… ▽ More

    Submitted 5 October, 2020; originally announced October 2020.

  14. arXiv:2009.11732  [pdf, other

    cs.LG cs.AI stat.ML

    A Unifying Review of Deep and Shallow Anomaly Detection

    Authors: Lukas Ruff, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, Klaus-Robert Müller

    Abstract: Deep learning approaches to anomaly detection have recently improved the state of the art in detection performance on complex datasets such as large collections of images or text. These results have sparked a renewed interest in the anomaly detection problem and led to the introduction of a great variety of new methods. With the emergence of numerous such methods, including approaches based on gen… ▽ More

    Submitted 8 February, 2021; v1 submitted 24 September, 2020; originally announced September 2020.

    Comments: 40 pages; accepted for publication in the Proceedings of the IEEE;

    Journal ref: Proceedings of the IEEE (2021) 1-40

  15. arXiv:2009.06571  [pdf, other

    cs.LG stat.ML

    Input Hessian Regularization of Neural Networks

    Authors: Waleed Mustafa, Robert A. Vandermeulen, Marius Kloft

    Abstract: Regularizing the input gradient has shown to be effective in promoting the robustness of neural networks. The regularization of the input's Hessian is therefore a natural next step. A key challenge here is the computational complexity. Computing the Hessian of inputs is computationally infeasible. In this paper we propose an efficient algorithm to train deep neural networks with Hessian operator-n… ▽ More

    Submitted 14 September, 2020; originally announced September 2020.

    Comments: Workshop on "Beyond first-order methods in ML systems" at the 37th International Conference on Machine Learning, Vienna, Austria, 2020

  16. arXiv:2007.01760  [pdf, other

    cs.CV cs.LG stat.ML

    Explainable Deep One-Class Classification

    Authors: Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, Klaus-Robert Müller

    Abstract: Deep one-class classification variants for anomaly detection learn a mapping that concentrates nominal samples in feature space causing anomalies to be mapped away. Because this transformation is highly non-linear, finding interpretations poses a significant challenge. In this paper we present an explainable deep one-class classification method, Fully Convolutional Data Description (FCDD), where t… ▽ More

    Submitted 18 March, 2021; v1 submitted 3 July, 2020; originally announced July 2020.

    Comments: 25 pages, published as a conference paper at ICLR 2021

  17. arXiv:2006.07459  [pdf, other

    stat.ML cs.LG math.ST

    Consistent Estimation of Identifiable Nonparametric Mixture Models from Grouped Observations

    Authors: Alexander Ritchie, Robert A. Vandermeulen, Clayton Scott

    Abstract: Recent research has established sufficient conditions for finite mixture models to be identifiable from grouped observations. These conditions allow the mixture components to be nonparametric and have substantial (or even total) overlap. This work proposes an algorithm that consistently estimates any identifiable mixture model from grouped observations. Our analysis leverages an oracle inequality… ▽ More

    Submitted 12 June, 2020; originally announced June 2020.

  18. arXiv:2006.00339  [pdf, other

    cs.LG stat.ML

    Rethinking Assumptions in Deep Anomaly Detection

    Authors: Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Klaus-Robert Müller, Marius Kloft

    Abstract: Though anomaly detection (AD) can be viewed as a classification problem (nominal vs. anomalous) it is usually treated in an unsupervised manner since one typically does not have access to, or it is infeasible to utilize, a dataset that sufficiently characterizes what it means to be "anomalous." In this paper we present results demonstrating that this intuition surprisingly seems not to extend to d… ▽ More

    Submitted 27 January, 2023; v1 submitted 30 May, 2020; originally announced June 2020.

    Comments: 17 pages; accepted at the ICML 2021 Workshop on Uncertainty & Robustness in Deep Learning; An extended Journal paper of this work has been published in Transactions on Machine Learning Research: arXiv:2205.11474

  19. arXiv:2001.10675  [pdf

    physics.chem-ph stat.ML

    Machine Learning in Thermodynamics: Prediction of Activity Coefficients by Matrix Completion

    Authors: Fabian Jirasek, Rodrigo A. S. Alves, Julie Damay, Robert A. Vandermeulen, Robert Bamler, Michael Bortz, Stephan Mandt, Marius Kloft, Hans Hasse

    Abstract: Activity coefficients, which are a measure of the non-ideality of liquid mixtures, are a key property in chemical engineering with relevance to modeling chemical and phase equilibria as well as transport processes. Although experimental data on thousands of binary mixtures are available, prediction methods are needed to calculate the activity coefficients in many relevant mixtures that have not be… ▽ More

    Submitted 28 January, 2020; originally announced January 2020.

    Comments: Published version: J. Phys. Chem. Lett. 11 (2020) 981-985; https://pubs.acs.org/doi/full/10.1021/acs.jpclett.9b03657

    Journal ref: J. Phys. Chem. Lett. 11 (2020) 981-985

  20. arXiv:1906.02694  [pdf, other

    cs.LG stat.ML

    Deep Semi-Supervised Anomaly Detection

    Authors: Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, Marius Kloft

    Abstract: Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anom… ▽ More

    Submitted 14 February, 2020; v1 submitted 6 June, 2019; originally announced June 2019.

    Comments: 23 pages, Published as a conference paper at ICLR 2020

  21. arXiv:1607.00071  [pdf, ps, other

    stat.ML math.ST

    An Operator Theoretic Approach to Nonparametric Mixture Models

    Authors: Robert A. Vandermeulen, Clayton D. Scott

    Abstract: When estimating finite mixture models, it is common to make assumptions on the mixture components, such as parametric assumptions. In this work, we make no distributional assumptions on the mixture components and instead assume that observations from the mixture model are grouped, such that observations in the same group are known to be drawn from the same mixture component. We precisely character… ▽ More

    Submitted 12 October, 2016; v1 submitted 30 June, 2016; originally announced July 2016.

    Comments: Contains and greatly extends the results from our previous work, arXiv:1502.06644, and thus contains some overlap with that work. This version contains some small grammatical and technical corrections as well as some changes for improved clarity

  22. arXiv:1502.06644  [pdf, ps, other

    stat.ML cs.LG math.ST

    On The Identifiability of Mixture Models from Grouped Samples

    Authors: Robert A. Vandermeulen, Clayton D. Scott

    Abstract: Finite mixture models are statistical models which appear in many problems in statistics and machine learning. In such models it is assumed that data are drawn from random probability measures, called mixture components, which are themselves drawn from a probability measure P over probability measures. When estimating mixture models, it is common to make assumptions on the mixture components, such… ▽ More

    Submitted 2 April, 2022; v1 submitted 23 February, 2015; originally announced February 2015.

    Comments: The work was subsumed and expanded upon in our Annals of Statistics publication "An Operator Theoretic Approach to Nonparametric Mixture Models."

  23. arXiv:1411.4378  [pdf, other

    stat.ML

    Robust Kernel Density Estimation by Scaling and Projection in Hilbert Space

    Authors: Robert A. Vandermeulen, Clayton D. Scott

    Abstract: While robust parameter estimation has been well studied in parametric density estimation, there has been little investigation into robust density estimation in the nonparametric setting. We present a robust version of the popular kernel density estimator (KDE). As with other estimators, a robust version of the KDE is useful since sample contamination is a common issue with datasets. What "robustne… ▽ More

    Submitted 17 November, 2014; originally announced November 2014.

    Comments: Extended version of NIPS 2014 paper