-
Data Augmentation and Regularization for Learning Group Equivariance
Authors:
Oskar Nordenfors,
Axel Flinth
Abstract:
In many machine learning tasks, known symmetries can be used as an inductive bias to improve model performance. In this paper, we consider learning group equivariance through training with data augmentation. We summarize results from a previous paper of our own, and extend the results to show that equivariance of the trained model can be achieved through training on augmented data in tandem with r…
▽ More
In many machine learning tasks, known symmetries can be used as an inductive bias to improve model performance. In this paper, we consider learning group equivariance through training with data augmentation. We summarize results from a previous paper of our own, and extend the results to show that equivariance of the trained model can be achieved through training on augmented data in tandem with regularization.
△ Less
Submitted 10 February, 2025;
originally announced February 2025.
-
Ensembles provably learn equivariance through data augmentation
Authors:
Oskar Nordenfors,
Axel Flinth
Abstract:
Recently, it was proved that group equivariance emerges in ensembles of neural networks as the result of full augmentation in the limit of infinitely wide neural networks (neural tangent kernel limit). In this paper, we extend this result significantly. We provide a proof that this emergence does not depend on the neural tangent kernel limit at all. We also consider stochastic settings, and furthe…
▽ More
Recently, it was proved that group equivariance emerges in ensembles of neural networks as the result of full augmentation in the limit of infinitely wide neural networks (neural tangent kernel limit). In this paper, we extend this result significantly. We provide a proof that this emergence does not depend on the neural tangent kernel limit at all. We also consider stochastic settings, and furthermore general architectures. For the latter, we provide a simple sufficient condition on the relation between the architecture and the action of the group for our results to hold. We validate our findings through simple numeric experiments.
△ Less
Submitted 2 October, 2024;
originally announced October 2024.
-
Optimization Dynamics of Equivariant and Augmented Neural Networks
Authors:
Oskar Nordenfors,
Fredrik Ohlsson,
Axel Flinth
Abstract:
We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show…
▽ More
We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models.
△ Less
Submitted 18 October, 2024; v1 submitted 23 March, 2023;
originally announced March 2023.
-
Grid is Good: Adaptive Refinement Algorithms for Off-the-Grid Total Variation Minimization
Authors:
Axel Flinth,
Frédéric de Gournay,
Pierre Weiss
Abstract:
We propose an adaptive refinement algorithm to solve total variation regularized measure optimization problems. The method iteratively constructs dyadic partitions of the unit cube based on i) the resolution of discretized dual problems and ii) on the detection of cells containing points that violate the dual constraints. The detection is based on upper-bounds on the dual certificate, in the spiri…
▽ More
We propose an adaptive refinement algorithm to solve total variation regularized measure optimization problems. The method iteratively constructs dyadic partitions of the unit cube based on i) the resolution of discretized dual problems and ii) on the detection of cells containing points that violate the dual constraints. The detection is based on upper-bounds on the dual certificate, in the spirit of branch-and-bound methods. The interest of this approach is that it avoids the use of heuristic approaches to find the maximizers of dual certificates. We prove the convergence of this approach under mild hypotheses and a linear convergence rate under additional non-degeneracy assumptions. These results are confirmed by simple numerical experiments.
△ Less
Submitted 18 January, 2023;
originally announced January 2023.
-
Bisparse Blind Deconvolution through Hierarchical Sparse Recovery
Authors:
Axel Flinth,
Ingo Roth,
Gerhard Wunder
Abstract:
The hierarchical sparsity framework, and in particular the HiHTP algorithm, has been successfully applied to many relevant communication engineering problems recently, particularly when the signal space is hierarchically structured. In this paper, the applicability of the HiHTP algorithm for solving the bi-sparse blind deconvolution problem is studied. The bi-sparse blind deconvolution setting her…
▽ More
The hierarchical sparsity framework, and in particular the HiHTP algorithm, has been successfully applied to many relevant communication engineering problems recently, particularly when the signal space is hierarchically structured. In this paper, the applicability of the HiHTP algorithm for solving the bi-sparse blind deconvolution problem is studied. The bi-sparse blind deconvolution setting here consists of recovering $h$ and $b$ from the knowledge of $h*(Qb)$, where $Q$ is some linear operator, and both $b$ and $h$ are both assumed to be sparse. The approach rests upon lifting the problem to a linear one, and then applying HiHTP, through the \emph{hierarchical sparsity framework}. %In particular, the efficient HiHTP algorithm is proposed for performing the recovery.
Then, for a Gaussian draw of the random matrix $Q$, it is theoretically shown that an $s$-sparse $h \in \mathbb{K}^μ$ and $σ$-sparse $b \in \mathbb{K}^n$ with high probability can be recovered when $μ\succcurlyeq s\log(s)^2\log(μ)\log(μn) + sσ\log(n)$.
△ Less
Submitted 10 November, 2024; v1 submitted 20 October, 2022;
originally announced October 2022.
-
Guaranteed blind deconvolution and demixing via hierarchically sparse reconstruction
Authors:
Axel Flinth,
Ingo Roth,
Benedikt Groß,
Jens Eisert,
Gerhard Wunder
Abstract:
The blind deconvolution problem amounts to reconstructing both a signal and a filter from the convolution of these two. It constitutes a prominent topic in mathematical and engineering literature. In this work, we analyze a sparse version of the problem: The filter $h\in \mathbb{R}^μ$ is assumed to be $s$-sparse, and the signal $b \in \mathbb{R}^n$ is taken to be $σ$-sparse, both supports being un…
▽ More
The blind deconvolution problem amounts to reconstructing both a signal and a filter from the convolution of these two. It constitutes a prominent topic in mathematical and engineering literature. In this work, we analyze a sparse version of the problem: The filter $h\in \mathbb{R}^μ$ is assumed to be $s$-sparse, and the signal $b \in \mathbb{R}^n$ is taken to be $σ$-sparse, both supports being unknown. We observe a convolution between the filter and a linear transformation of the signal. Motivated by practically important multi-user communication applications, we derive a recovery guarantee for the simultaneous demixing and deconvolution setting. We achieve efficient recovery by relaxing the problem to a hierarchical sparse recovery for which we can build on a flexible framework. At the same time, for this we pay the price of some sub-optimal guarantees compared to the number of free parameters of the problem. The signal model we consider is sufficiently general to capture many applications in a number of engineering fields. Despite their practical importance, we provide first rigorous performance guarantees for efficient and simple algorithms for the bi-sparse and generalized demixing setting. We complement our analytical results by presenting results of numerical simulations. We find evidence that the sub-optimal scaling $s^2σ\log(μ)\log(n)$ of our derived sufficient condition is likely overly pessimistic and that the observed performance is better described by a scaling proportional to $ sσ$ up to log-factors.
△ Less
Submitted 5 November, 2021;
originally announced November 2021.
-
Hierarchical Isometry Properties of Hierarchical Measurements
Authors:
Axel Flinth,
Benedikt Groß,
Ingo Roth,
Jens Eisert,
Gerhard Wunder
Abstract:
A new class of measurement operators, coined hierarchical measurement operators, and prove results guaranteeing the efficient, stable and robust recovery of hierarchically structured signals from such measurements. We derive bounds on their hierarchical restricted isometry properties based on the restricted isometry constants of their constituent matrices, generalizing and extending prior work on…
▽ More
A new class of measurement operators, coined hierarchical measurement operators, and prove results guaranteeing the efficient, stable and robust recovery of hierarchically structured signals from such measurements. We derive bounds on their hierarchical restricted isometry properties based on the restricted isometry constants of their constituent matrices, generalizing and extending prior work on Kronecker-product measurements. As an exemplary application, we apply the theory to two communication scenarios. The fast and scalable HiHTP algorithm is shown to be suitable for solving these types of problems and its performance is evaluated numerically in terms of sparse signal recovery and block detection capability.
△ Less
Submitted 14 December, 2021; v1 submitted 20 May, 2020;
originally announced May 2020.
-
On the linear convergence rates of exchange and continuous methods for total variation minimization
Authors:
Axel Flinth,
Frédéric de Gournay,
Pierre Weiss
Abstract:
We analyze an exchange algorithm for the numerical solution total-variation regularized inverse problems over the space M($Ω$) of Radon measures on a subset $Ω$ of R d. Our main result states that under some regularity conditions, the method eventually converges linearly. Additionally, we prove that continuously optimizing the amplitudes of positions of the target measure will succeed at a linear…
▽ More
We analyze an exchange algorithm for the numerical solution total-variation regularized inverse problems over the space M($Ω$) of Radon measures on a subset $Ω$ of R d. Our main result states that under some regularity conditions, the method eventually converges linearly. Additionally, we prove that continuously optimizing the amplitudes of positions of the target measure will succeed at a linear rate with a good initialization. Finally, we propose to combine the two approaches into an alternating method and discuss the comparative advantages of this approach.
△ Less
Submitted 24 June, 2019;
originally announced June 2019.
-
Compressed Sensing for Analog Signals
Authors:
Bernard G. Bodmann,
Axel Flinth,
Gitta Kutyniok
Abstract:
In this paper we develop a general theory of compressed sensing for analog signals, in close similarity to prior results for vectors in finite dimensional spaces that are sparse in a given orthonormal basis. The signals are modeled by functions in a reproducing kernel Hilbert space. Sparsity is defined as the minimal number of terms in expansions based on the kernel functions. Minimizing this numb…
▽ More
In this paper we develop a general theory of compressed sensing for analog signals, in close similarity to prior results for vectors in finite dimensional spaces that are sparse in a given orthonormal basis. The signals are modeled by functions in a reproducing kernel Hilbert space. Sparsity is defined as the minimal number of terms in expansions based on the kernel functions. Minimizing this number is under certain conditions equivalent to minimizing an atomic norm, the pre-dual of the supremum norm for functions in the Hilbert space. The norm minimizer is shown to exist based on a compactness argument. Recovery based on minimizing the atomic norm is robust and stable, so it provides controllable accuracy for recovery when the signal is only approximately sparse and the measurement is corrupted by noise.
As applications of the theory, we include results on the recovery of sparse bandlimited functions and functions that have a sparse inverse short-time Fourier transform.
△ Less
Submitted 12 March, 2018;
originally announced March 2018.
-
Recovery of Binary Sparse Signals with Biased Measurement Matrices
Authors:
Axel Flinth,
Sandra Keiper
Abstract:
This work treats the recovery of sparse, binary signals through box-constrained basis pursuit using biased measurement matrices. Using a probabilistic model, we provide conditions under which the recovery of both sparse and saturated binary signals is very likely. In fact, we also show that under the same condition, the solution of the boxed-constrained basis pursuit program can be found using box…
▽ More
This work treats the recovery of sparse, binary signals through box-constrained basis pursuit using biased measurement matrices. Using a probabilistic model, we provide conditions under which the recovery of both sparse and saturated binary signals is very likely. In fact, we also show that under the same condition, the solution of the boxed-constrained basis pursuit program can be found using boxed-constrained least-squares.
△ Less
Submitted 10 January, 2018;
originally announced January 2018.
-
Thermal Source Localization Through Infinite-Dimensional Compressed Sensing
Authors:
Axel Flinth,
Ali Hashemi
Abstract:
We propose a scheme utilizing ideas from infinite dimensional compressed sensing for thermal source localization. Using the soft recovery framework of one of the authors, we provide rigorous theoretical guarantees for the recovery performance. In particular, we extend the framework in order to also include noisy measurements. Further, we conduct numerical experiments, showing that our proposed met…
▽ More
We propose a scheme utilizing ideas from infinite dimensional compressed sensing for thermal source localization. Using the soft recovery framework of one of the authors, we provide rigorous theoretical guarantees for the recovery performance. In particular, we extend the framework in order to also include noisy measurements. Further, we conduct numerical experiments, showing that our proposed method has strong performance, in a wide range of settings. These include scenarios with few sensors, off-grid source positioning and high noise levels, both in one and two dimensions.
△ Less
Submitted 4 October, 2017;
originally announced October 2017.
-
Exact solutions of infinite dimensional total-variation regularized problems
Authors:
Axel Flinth,
Pierre Weiss
Abstract:
We study the solutions of infinite dimensional linear inverse problems over Banach spaces. The regularizer is defined as the total variation of a linear mapping of the function to recover, while the data fitting term is a near arbitrary convex function. The first contribution is about the solu-tion's structure: we show that under suitable assumptions, there always exist an m-sparse solution, where…
▽ More
We study the solutions of infinite dimensional linear inverse problems over Banach spaces. The regularizer is defined as the total variation of a linear mapping of the function to recover, while the data fitting term is a near arbitrary convex function. The first contribution is about the solu-tion's structure: we show that under suitable assumptions, there always exist an m-sparse solution, where m is the number of linear measurements of the signal. Our second contribution is about the computation of the solution. While most existing works first discretize the problem, we show that exacts solutions of the infinite dimensional problem can be obtained by solving two consecutive finite dimensional convex programs. These results extend recent advances in the understanding of total-variation reg-ularized problems.
△ Less
Submitted 2 November, 2017; v1 submitted 7 August, 2017;
originally announced August 2017.
-
Soft Recovery With General Atomic Norms
Authors:
Axel Flinth
Abstract:
This paper describes a dual certificate condition on a linear measurement operator $A$ (defined on a Hilbert space $\mathcal{H}$ and having finite-dimensional range) which guarantees that an atomic norm minimization, in a certain sense, will be able to approximately recover a structured signal $v_0 \in \mathcal{H}$ from measurements $Av_0$. Put very streamlined, the condition implies that peaks in…
▽ More
This paper describes a dual certificate condition on a linear measurement operator $A$ (defined on a Hilbert space $\mathcal{H}$ and having finite-dimensional range) which guarantees that an atomic norm minimization, in a certain sense, will be able to approximately recover a structured signal $v_0 \in \mathcal{H}$ from measurements $Av_0$. Put very streamlined, the condition implies that peaks in a sparse decomposition of $v_0$ are close the the support of the atomic decomposition of the solution $v^*$. The condition applies in a relatively general context - in particular, the space $\mathcal{H}$ can be infinite-dimensional. The abstract framework is applied to several concrete examples, one example being super-resolution. In this process, several novel results which are interesting on its own are obtained.
△ Less
Submitted 10 May, 2017;
originally announced May 2017.
-
Sparse Blind Deconvolution and Demixing Through $\ell_{1,2}$-Minimization
Authors:
Axel Flinth
Abstract:
This paper concerns solving the sparse deconvolution and demixing problem using $\ell_{1,2}$-minimization. We show that under a certain structured random model, robust and stable recovery is possible. The results extend results of Ling and Strohmer [Self Calibration and Biconvex Compressive Sensing, Inverse Problems, 2015], and in particular theoretically explain certain experimental findings from…
▽ More
This paper concerns solving the sparse deconvolution and demixing problem using $\ell_{1,2}$-minimization. We show that under a certain structured random model, robust and stable recovery is possible. The results extend results of Ling and Strohmer [Self Calibration and Biconvex Compressive Sensing, Inverse Problems, 2015], and in particular theoretically explain certain experimental findings from that paper. Our results do not only apply to the deconvolution and demixing problem, but to recovery of column-sparse matrices in general.
△ Less
Submitted 13 April, 2017; v1 submitted 8 September, 2016;
originally announced September 2016.
-
Soft Recovery Through $\ell_{1,2}$ Minimization with Applications in Recovery of Simultaneously Sparse and Low-Rank Matrice
Authors:
Axel Flinth
Abstract:
This article provides a new type of analysis of a compressed-sensing based technique for recovering column-sparse matrices, namely minimization of the $\ell_{1,2}$-norm. Rather than providing conditions on the measurement matrix which guarantees the solution of the program to be exactly equal to the ground truth signal (which already has been thoroughly investigated), it presents a condition which…
▽ More
This article provides a new type of analysis of a compressed-sensing based technique for recovering column-sparse matrices, namely minimization of the $\ell_{1,2}$-norm. Rather than providing conditions on the measurement matrix which guarantees the solution of the program to be exactly equal to the ground truth signal (which already has been thoroughly investigated), it presents a condition which guarantees that the solution is approximately equal to the ground truth. Soft recovery statements of this kind are to the best knowledge of the author a novelty in Compressed Sensing. Apart from the theoretical analysis, we present two heuristic proposes how this property of the $\ell_{1,2}$-program can be utilized to design algorithms for recovery of matrices which are sparse and have low rank at the same time.
△ Less
Submitted 8 September, 2016;
originally announced September 2016.
-
A Geometrical Stability Condition for Compressed Sensing
Authors:
Axel Flinth
Abstract:
During the last decade, the paradigm of compressed sensing has gained significant importance in the signal processing community. While the original idea was to utilize sparsity assumptions to design powerful recovery algorithms of vectors $x \in \mathbb{R}^d$, the concept has been extended to cover many other types of problems. A noteable example is low-rank matrix recovery. Many methods used for…
▽ More
During the last decade, the paradigm of compressed sensing has gained significant importance in the signal processing community. While the original idea was to utilize sparsity assumptions to design powerful recovery algorithms of vectors $x \in \mathbb{R}^d$, the concept has been extended to cover many other types of problems. A noteable example is low-rank matrix recovery. Many methods used for recovery rely on solving convex programs.
A particularly nice trait of compressed sensing is its geometrical intuition. In recent papers, a classical optimality condition has been used together with tools from convex geometry and probability theory to prove beautiful results concerning the recovery of signals from Gaussian measurements. In this paper, we aim to formulate a geometrical condition for stability and robustness, i.e. for the recovery of approximately structured signals from noisy measurements.
We will investigate the connection between the new condition with the notion of restricted singular values, classical stability and robustness conditions in compressed sensing, and also to important geometrical concepts from complexity theory. We will also prove the maybe somewhat surprising fact that for many convex programs, exact recovery of a signal $x_0$ immediately implies some stability and robustness when recovering signals close to $x_0$.
△ Less
Submitted 6 July, 2016; v1 submitted 28 October, 2015;
originally announced October 2015.
-
Optimal Choice of Weights for Sparse Recovery With Prior Information
Authors:
Axel Flinth
Abstract:
Compressed sensing deals with the recovery of sparse signals from linear measurements. Without any additional information, it is possible to recover an $s$-sparse signal using $m \gtrsim s \log(d/s)$ measurements in a robust and stable way. Some applications provide additional information, such as on the location of the support of the signal. Using this information, it is conceivable the threshold…
▽ More
Compressed sensing deals with the recovery of sparse signals from linear measurements. Without any additional information, it is possible to recover an $s$-sparse signal using $m \gtrsim s \log(d/s)$ measurements in a robust and stable way. Some applications provide additional information, such as on the location of the support of the signal. Using this information, it is conceivable the threshold amount of measurements can be lowered. A proposed algorithm for this task is \emph{weighted $\ell_1$-minimization}. Put shortly, one modifies standard $\ell_1$-minimization by assigning different weights to different parts of the index set $[1, \dots d]$. The task of choosing the weights is however non-trivial.
This paper provides a complete answer to the question of an optimal choice of the weights. In fact, it is shown that it is possible to directly calculate unique weights that are optimal in the sense that the threshold amount of measurements needed for exact recovery is minimized. The proof uses recent results about the connection between convex geometry and compressed sensing-type algorithms.
△ Less
Submitted 24 May, 2016; v1 submitted 30 June, 2015;
originally announced June 2015.
-
Multivariate $α$-molecules
Authors:
Axel Flinth,
Martin Schäfer
Abstract:
The suboptimal performance of wavelets with regard to the approximation of multivariate data gave rise to new representation systems, specifically designed for data with anisotropic features. Some prominent examples of these are given by ridgelets, curvelets, and shearlets, to name a few.
The great variety of such so-called directional systems motivated the search for a common framework, which u…
▽ More
The suboptimal performance of wavelets with regard to the approximation of multivariate data gave rise to new representation systems, specifically designed for data with anisotropic features. Some prominent examples of these are given by ridgelets, curvelets, and shearlets, to name a few.
The great variety of such so-called directional systems motivated the search for a common framework, which unites many under one roof and enables a simultaneous analysis, for example with respect to approximation properties. Building on the concept of parabolic molecules, the recently introduced framework of $α$-molecules does in fact include the previous mentioned systems. Until now however it is confined to the bivariate setting, whereas nowadays one often deals with higher dimensional data. This motivates the extension of this unifying theory to dimensions larger than 2, put forward in this work. In particular, we generalize the central result that the cross-Gramian of any two systems of $α$-molecules will to some extent be localized.
As an exemplary application, we investigate the sparse approximation of video signals, which are instances of 3D data. The multivariate theory allows us to derive almost optimal approximation rates for a large class of representation systems.
△ Less
Submitted 8 January, 2016; v1 submitted 27 April, 2015;
originally announced April 2015.
-
Phase Retrieval from Gabor Measurements
Authors:
Irena Bojarovska,
Axel Flinth
Abstract:
Compressed sensing investigates the recovery of sparse signals from linear measurements. But often, in a wide range of applications, one is given only the absolute values (squared) of the linear measurements. Recovering such signals (not necessarily sparse) is known as the phase retrieval problem. We consider this problem in the case when the measurements are time-frequency shifts of a suitably ch…
▽ More
Compressed sensing investigates the recovery of sparse signals from linear measurements. But often, in a wide range of applications, one is given only the absolute values (squared) of the linear measurements. Recovering such signals (not necessarily sparse) is known as the phase retrieval problem. We consider this problem in the case when the measurements are time-frequency shifts of a suitably chosen generator, i.e. coming from a Gabor frame. We prove an easily checkable injectivity condition for recovery of any signal from all $N^2$ time-frequency shifts, and for recovery of sparse signals, when only some of those measurements are given.
△ Less
Submitted 28 September, 2015; v1 submitted 19 March, 2015;
originally announced March 2015.
