-
Convergence of alternating minimisation algorithms for dictionary learning
Authors:
Simon Ruetz,
Karin Schnass
Abstract:
In this paper we derive sufficient conditions for the convergence of two popular alternating minimisation algorithms for dictionary learning - the Method of Optimal Directions (MOD) and Online Dictionary Learning (ODL), which can also be thought of as approximative K-SVD. We show that given a well-behaved initialisation that is either within distance at most $1/\log(K)$ to the generating dictionar…
▽ More
In this paper we derive sufficient conditions for the convergence of two popular alternating minimisation algorithms for dictionary learning - the Method of Optimal Directions (MOD) and Online Dictionary Learning (ODL), which can also be thought of as approximative K-SVD. We show that given a well-behaved initialisation that is either within distance at most $1/\log(K)$ to the generating dictionary or has a special structure ensuring that each element of the initialisation only points to one generating element, both algorithms will converge with geometric convergence rate to the generating dictionary. This is done even for data models with non-uniform distributions on the supports of the sparse coefficients. These allow the appearance frequency of the dictionary elements to vary heavily and thus model real data more closely.
△ Less
Submitted 26 May, 2023; v1 submitted 4 April, 2023;
originally announced April 2023.
-
Non-asymptotic bounds for inclusion probabilities in rejective sampling
Authors:
Simon Ruetz,
Karin Schnass
Abstract:
We provide non-asymptotic bounds for first and higher order inclusion probabilities of the rejective sampling model with various size parameters. Further we derive bounds in the semi-definite ordering for matrices that collect (conditional) first and second order inclusion probabilities as their diagonal resp. off-diagonal entries.
We provide non-asymptotic bounds for first and higher order inclusion probabilities of the rejective sampling model with various size parameters. Further we derive bounds in the semi-definite ordering for matrices that collect (conditional) first and second order inclusion probabilities as their diagonal resp. off-diagonal entries.
△ Less
Submitted 19 December, 2022;
originally announced December 2022.
-
Adapted variable density subsampling for compressed sensing
Authors:
Simon Ruetz
Abstract:
Recent results in compressed sensing showed that the optimal subsampling strategy should take into account the sparsity pattern of the signal at hand. This oracle-like knowledge, even though desirable, nevertheless remains elusive in most practical application. We try to close this gap by showing how the sparsity patterns can instead be characterised via a probability distribution on the supports…
▽ More
Recent results in compressed sensing showed that the optimal subsampling strategy should take into account the sparsity pattern of the signal at hand. This oracle-like knowledge, even though desirable, nevertheless remains elusive in most practical application. We try to close this gap by showing how the sparsity patterns can instead be characterised via a probability distribution on the supports of the sparse signals allowing us to again derive optimal subsampling strategies. This probability distribution can be easily estimated from signals of the same signal class, achieving state of the art performance in numerical experiments. Our approach also extends to structured acquisition, where instead of isolated measurements, blocks of measurements are taken.
△ Less
Submitted 26 June, 2023; v1 submitted 28 June, 2022;
originally announced June 2022.
-
Fast Single-Core K-Nearest Neighbor Graph Computation
Authors:
Dan Kluser,
Jonas Bokstaller,
Samuel Rutz,
Tobias Buner
Abstract:
Fast and reliable K-Nearest Neighbor Graph algorithms are more important than ever due to their widespread use in many data processing techniques. This paper presents a runtime optimized C implementation of the heuristic "NN-Descent" algorithm by Wei Dong et al. for the l2-distance metric. Various implementation optimizations are explained which improve performance for low-dimensional as well as h…
▽ More
Fast and reliable K-Nearest Neighbor Graph algorithms are more important than ever due to their widespread use in many data processing techniques. This paper presents a runtime optimized C implementation of the heuristic "NN-Descent" algorithm by Wei Dong et al. for the l2-distance metric. Various implementation optimizations are explained which improve performance for low-dimensional as well as high dimensional datasets. Optimizations to speed up the selection of which datapoint pairs to evaluate the distance for are primarily impactful for low-dimensional datasets. A heuristic which exploits the iterative nature of NN-Descent to reorder data in memory is presented which enables better use of locality and thereby improves the runtime. The restriction to the l2-distance metric allows for the use of blocked distance evaluations which significantly increase performance for high dimensional datasets. In combination the optimizations yield an implementation which significantly outperforms a widely used implementation of NN-Descent on all considered datasets. For instance, the runtime on the popular MNIST handwritten digits dataset is halved.
△ Less
Submitted 13 December, 2021;
originally announced December 2021.
-
Modeling Coral Reef Bleaching Recovery Through KCC-Theory
Authors:
Rafael Cavalcanti,
Solange Rutz
Abstract:
We use Volterra-Hamilton systems theory and their associated cost functional to study the population dynamics and productive processes of coral reefs in recovery from bleaching and show that the cost of production remains the same after the process. The KCC-theory geometrical invariants are determined for the model proposed to describe the renewed symbiotic interaction between coral and algae.
We use Volterra-Hamilton systems theory and their associated cost functional to study the population dynamics and productive processes of coral reefs in recovery from bleaching and show that the cost of production remains the same after the process. The KCC-theory geometrical invariants are determined for the model proposed to describe the renewed symbiotic interaction between coral and algae.
△ Less
Submitted 19 March, 2021;
originally announced March 2021.
-
Submatrices with non-uniformly selected random supports and insights into sparse approximation
Authors:
Simon Ruetz,
Karin Schnass
Abstract:
In this paper we derive tail bounds on the norms of random submatrices with non-uniformly distributed supports. We apply these results to sparse approximation and conduct an analysis of the average case performance of thresholding, Orthogonal Matching Pursuit and Basis Pursuit. As an application of these results we characterise sensing dictionaries to improve average performance in the non-uniform…
▽ More
In this paper we derive tail bounds on the norms of random submatrices with non-uniformly distributed supports. We apply these results to sparse approximation and conduct an analysis of the average case performance of thresholding, Orthogonal Matching Pursuit and Basis Pursuit. As an application of these results we characterise sensing dictionaries to improve average performance in the non-uniform case and test their performance numerically.
△ Less
Submitted 3 December, 2020;
originally announced December 2020.
