Showing 1–35 of 35 results for author: Usevich, K

Personalized Coupled Tensor Decomposition for Multimodal Data Fusion: Uniqueness and Algorithms

Authors: Ricardo Augusto Borsoi, Konstantin Usevich, David Brie, Tülay Adali

Abstract: Coupled tensor decompositions (CTDs) perform data fusion by linking factors from different datasets. Although many CTDs have been already proposed, current works do not address important challenges of data fusion, where: 1) the datasets are often heterogeneous, constituting different "views" of a given phenomena (multimodality); and 2) each dataset can contain personalized or dataset-specific info… ▽ More Coupled tensor decompositions (CTDs) perform data fusion by linking factors from different datasets. Although many CTDs have been already proposed, current works do not address important challenges of data fusion, where: 1) the datasets are often heterogeneous, constituting different "views" of a given phenomena (multimodality); and 2) each dataset can contain personalized or dataset-specific information, constituting distinct factors that are not coupled with other datasets. In this work, we introduce a personalized CTD framework tackling these challenges. A flexible model is proposed where each dataset is represented as the sum of two components, one related to a common tensor through a multilinear measurement model, and another specific to each dataset. Both the common and distinct components are assumed to admit a polyadic decomposition. This generalizes several existing CTD models. We provide conditions for specific and generic uniqueness of the decomposition that are easy to interpret. These conditions employ uni-mode uniqueness of different individual datasets and properties of the measurement model. Two algorithms are proposed to compute the common and distinct components: a semi-algebraic one and a coordinate-descent optimization method. Experimental results illustrate the advantage of the proposed framework compared with the state of the art approaches. △ Less

Submitted 12 December, 2024; v1 submitted 1 December, 2024; originally announced December 2024.

Computing asymptotic eigenvectors and eigenvalues of perturbed symmetric matrices

Authors: Konstantin Usevich, Simon Barthelme

Abstract: Computing the eigenvectors and eigenvalues of a perturbed matrix can be remarkably difficult when the unperturbed matrix has repeated eigenvalues. In this work we show how the limiting eigenvectors and eigenvalues of a symmetric matrix $K(\varepsilon)$ as $\varepsilon \to 0$ can be obtained relatively easily from successive Schur complements, provided that the entries scale in different orders of… ▽ More Computing the eigenvectors and eigenvalues of a perturbed matrix can be remarkably difficult when the unperturbed matrix has repeated eigenvalues. In this work we show how the limiting eigenvectors and eigenvalues of a symmetric matrix $K(\varepsilon)$ as $\varepsilon \to 0$ can be obtained relatively easily from successive Schur complements, provided that the entries scale in different orders of $\varepsilon$. If the matrix does not directly exhibit this structure, we show that putting the matrix into a ``generalised kernel form'' can be very informative. The resulting formulas are much simpler than classical expressions obtained from complex integrals involving the resolvent. We apply our results to the problem of computing the eigenvalues and eigenvectors of kernel matrices in the ``flat limit'', a problem that appears in many applications in statistics and approximation theory. In particular, we prove a conjecture from [SIAM J. Matrix Anal. Appl., 2021, 42(1):17--57] which connects the eigenvectors of kernel matrices to multivariate orthogonal polynomials. △ Less

Submitted 24 July, 2024; originally announced July 2024.

On factorization of rank-one auto-correlation matrix polynomials

Authors: Konstantin Usevich, Julien Flamant, Marianne Clausel, David Brie

Abstract: This article characterizes the rank-one factorization of auto-correlation matrix polynomials. We establish a sufficient and necessary uniqueness condition for uniqueness of the factorization based on the greatest common divisor (GCD) of multiple polynomials. In the unique case, we show that the factorization can be carried out explicitly using GCDs. In the non-unique case, the number of non-triv… ▽ More This article characterizes the rank-one factorization of auto-correlation matrix polynomials. We establish a sufficient and necessary uniqueness condition for uniqueness of the factorization based on the greatest common divisor (GCD) of multiple polynomials. In the unique case, we show that the factorization can be carried out explicitly using GCDs. In the non-unique case, the number of non-trivially different factorizations is given and all solutions are enumerated. △ Less

Submitted 29 August, 2023; originally announced August 2023.

Report number: BioSiS

The barycenter in free nilpotent Lie groups and its application to iterated-integrals signatures

Authors: Marianne Clausel, Joscha Diehl, Raphael Mignot, Leonard Schmitz, Nozomi Sugiura, Konstantin Usevich

Abstract: We establish the well-definedness of the barycenter (in the sense of Buser and Karcher) for every integrable measure on the free nilpotent Lie group of step $L$ (over $\mathbb{R}^d$). We provide two algorithms for computing it, using methods from Lie theory (namely, the Baker-Campbell-Hausdorff formula) and from the theory of Gröbner bases of modules. Our main motivation stems from measures induce… ▽ More We establish the well-definedness of the barycenter (in the sense of Buser and Karcher) for every integrable measure on the free nilpotent Lie group of step $L$ (over $\mathbb{R}^d$). We provide two algorithms for computing it, using methods from Lie theory (namely, the Baker-Campbell-Hausdorff formula) and from the theory of Gröbner bases of modules. Our main motivation stems from measures induced by iterated-integrals signatures, and we calculate the barycenter for the signature of the Brownian motion. △ Less

Submitted 9 January, 2024; v1 submitted 30 May, 2023; originally announced May 2023.

Comments: 48 pages, 1 figure

MSC Class: 60L10 (Primary) 22E25; 60J65; 13P10; 15A69 (Secondary)

A lifting approach to ParaTuck-2 tensor decompositions

Authors: Konstantin Usevich

Abstract: The ParaTuck-2 decomposition (PT2D) of third-order tensor is a two-layer generalization of the well-known canonical polyadic decomposition (CPD).While being more flexible than the CPD, the PT2D also possesses similar uniqueness properties.In this paper, we show than under the best known uniqueness conditions, the exact PT2D can be computed by an algebraic algorithm (i.e., can the PT2D problems can… ▽ More The ParaTuck-2 decomposition (PT2D) of third-order tensor is a two-layer generalization of the well-known canonical polyadic decomposition (CPD).While being more flexible than the CPD, the PT2D also possesses similar uniqueness properties.In this paper, we show than under the best known uniqueness conditions, the exact PT2D can be computed by an algebraic algorithm (i.e., can the PT2D problems can be reduced to computing nullspaces and eigenvalues of certain matrices).We do so by lifting the slices of the tensor to higher-dimensional space, which also allows for refining the existing uniqueness conditions.The algorithms are developed for general PT2D and its symmetric version (DEDICOM), which leads to an algebraic algorithm for another generalization of the CPD, the PARAFAC2 decomposition.Our methods are also applicable in the approximation scenario, as shown by the numerical experiments. △ Less

Submitted 10 March, 2025; v1 submitted 2 February, 2023; originally announced February 2023.

Coupled CP tensor decomposition with shared and distinct components for multi-task fMRI data fusion

Authors: Ricardo Augusto Borsoi, Isabell Lehmann, Mohammad Abu Baker Siddique Akhonda, Vince Calhoun, Konstantin Usevich, David Brie, Tülay Adali

Abstract: Discovering components that are shared in multiple datasets, next to dataset-specific features, has great potential for studying the relationships between different subjects or tasks in functional Magnetic Resonance Imaging (fMRI) data. Coupled matrix and tensor factorization approaches have been useful for flexible data fusion, or decomposition to extract features that can be used in multiple way… ▽ More Discovering components that are shared in multiple datasets, next to dataset-specific features, has great potential for studying the relationships between different subjects or tasks in functional Magnetic Resonance Imaging (fMRI) data. Coupled matrix and tensor factorization approaches have been useful for flexible data fusion, or decomposition to extract features that can be used in multiple ways. However, existing methods do not directly recover shared and dataset-specific components, which requires post-processing steps involving additional hyperparameter selection. In this paper, we propose a tensor-based framework for multi-task fMRI data fusion, using a partially constrained canonical polyadic (CP) decomposition model. Differently from previous approaches, the proposed method directly recovers shared and dataset-specific components, leading to results that are directly interpretable. A strategy to select a highly reproducible solution to the decomposition is also proposed. We evaluate the proposed methodology on real fMRI data of three tasks, and show that the proposed method finds meaningful components that clearly identify group differences between patients with schizophrenia and healthy controls. △ Less

Submitted 23 July, 2023; v1 submitted 25 November, 2022; originally announced November 2022.

Polarimetric phase retrieval: uniqueness and algorithms

Authors: Julien Flamant, Konstantin Usevich, Marianne Clausel, David Brie

Abstract: This work introduces a novel Fourier phase retrieval model, called polarimetric phase retrieval that enables a systematic use of polarization information in Fourier phase retrieval problems. We provide a complete characterization of uniqueness properties of this new model by unraveling equivalencies with a peculiar polynomial factorization problem. We introduce two different but complementary cate… ▽ More This work introduces a novel Fourier phase retrieval model, called polarimetric phase retrieval that enables a systematic use of polarization information in Fourier phase retrieval problems. We provide a complete characterization of uniqueness properties of this new model by unraveling equivalencies with a peculiar polynomial factorization problem. We introduce two different but complementary categories of reconstruction methods. The first one is algebraic and relies on the use of approximate greatest common divisor computations using Sylvester matrices. The second one carefully adapts existing algorithms for Fourier phase retrieval, namely semidefinite positive relaxation and Wirtinger-Flow, to solve the polarimetric phase retrieval problem. Finally, a set of numerical experiments permits a detailed assessment of the numerical behavior and relative performances of each proposed reconstruction strategy. We further highlight a reconstruction strategy that combines both approaches for scalable, computationally efficient and asymptotically MSE optimal performance. △ Less

Submitted 26 June, 2022; originally announced June 2022.

Comments: 37 pages, 10 figures

Hankel low-rank approximation and completion in time series analysis and forecasting: a brief review

Authors: Jonathan Gillard, Konstantin Usevich

Abstract: In this paper we offer a review and bibliography of work on Hankel low-rank approximation and completion, with particular emphasis on how this methodology can be used for time series analysis and forecasting. We begin by describing possible formulations of the problem and offer commentary on related topics and challenges in obtaining globally optimal solutions. Key theorems are provided, and the p… ▽ More In this paper we offer a review and bibliography of work on Hankel low-rank approximation and completion, with particular emphasis on how this methodology can be used for time series analysis and forecasting. We begin by describing possible formulations of the problem and offer commentary on related topics and challenges in obtaining globally optimal solutions. Key theorems are provided, and the paper closes with some expository examples. △ Less

Submitted 10 June, 2022; originally announced June 2022.

Report number: BioSiS

Journal ref: Statistics and Its Interface, International Press, In press

Gaussian Process Regression in the Flat Limit

Authors: Simon Barthelmé, Pierre-Olivier Amblard, Nicolas Tremblay, Konstantin Usevich

Abstract: Gaussian process (GP) regression is a fundamental tool in Bayesian statistics. It is also known as kriging and is the Bayesian counterpart to the frequentist kernel ridge regression. Most of the theoretical work on GP regression has focused on a large-$n$ asymptotics, i.e. as the amount of data increases. Fixed-sample analysis is much more difficult outside of simple cases, such as locations on a… ▽ More Gaussian process (GP) regression is a fundamental tool in Bayesian statistics. It is also known as kriging and is the Bayesian counterpart to the frequentist kernel ridge regression. Most of the theoretical work on GP regression has focused on a large-$n$ asymptotics, i.e. as the amount of data increases. Fixed-sample analysis is much more difficult outside of simple cases, such as locations on a regular grid. In this work we perform a fixed-sample analysis that was first studied in the context of approximation theory by Driscoll & Fornberg (2002), called the ``flat limit''. In flat-limit asymptotics, the goal is to characterise kernel methods as the length-scale of the kernel function tends to infinity, so that kernels appear flat over the range of the data. Surprisingly, this limit is well-defined, and displays interesting behaviour: Driscoll & Fornberg showed that radial basis interpolation converges in the flat limit to polynomial interpolation, if the kernel is Gaussian. Subsequent work showed that this holds true in the multivariate setting as well, but that kernels other than the Gaussian may have (polyharmonic) splines as the limit interpolant. Leveraging recent results on the spectral behaviour of kernel matrices in the flat limit, we study the flat limit of Gaussian process regression. Results show that Gaussian process regression tends in the flat limit to (multivariate) polynomial regression, or (polyharmonic) spline regression, depending on the kernel. Importantly, this holds for both the predictive mean and the predictive variance, so that the posterior predictive distributions become equivalent. Our results have practical consequences: for instance, they show that optimal GP predictions in the sense of leave-one-out loss may occur at very large length-scales, which would be invisible to current implementations because of numerical difficulties. △ Less

Submitted 26 October, 2023; v1 submitted 4 January, 2022; originally announced January 2022.

Robust Eigenvectors of Symmetric Tensors

Authors: Tommi Muller, Elina Robeva, Konstantin Usevich

Abstract: The tensor power method generalizes the matrix power method to higher order arrays, or tensors. Like in the matrix case, the fixed points of the tensor power method are the eigenvectors of the tensor. While every real symmetric matrix has an eigendecomposition, the vectors generating a symmetric decomposition of a real symmetric tensor are not always eigenvectors of the tensor. In this paper we… ▽ More The tensor power method generalizes the matrix power method to higher order arrays, or tensors. Like in the matrix case, the fixed points of the tensor power method are the eigenvectors of the tensor. While every real symmetric matrix has an eigendecomposition, the vectors generating a symmetric decomposition of a real symmetric tensor are not always eigenvectors of the tensor. In this paper we show that whenever an eigenvector is a generator of the symmetric decomposition of a symmetric tensor, then (if the order of the tensor is sufficiently high) this eigenvector is robust, i.e., it is an attracting fixed point of the tensor power method. We exhibit new classes of symmetric tensors whose symmetric decomposition consists of eigenvectors. Generalizing orthogonally decomposable tensors, we consider equiangular tight frame decomposable and equiangular set decomposable tensors. Our main result implies that such tensors can be decomposed using the tensor power method. △ Less

Submitted 27 March, 2025; v1 submitted 12 November, 2021; originally announced November 2021.

Comments: 22 pages, 3 figures

MSC Class: 15A69; 15A18; 42C15; 65F15

Journal ref: SIAM J. Matrix Anal. Appl., 43(4):1784--1805, 2022

Low-rank tensor recovery for Jacobian-based Volterra identification of parallel Wiener-Hammerstein systems

Authors: Konstantin Usevich, Philippe Dreesen, Mariya Ishteva

Abstract: We consider the problem of identifying a parallel Wiener-Hammerstein structure from Volterra kernels. Methods based on Volterra kernels typically resort to coupled tensor decompositions of the kernels. However, in the case of parallel Wiener-Hammerstein systems, such methods require nontrivial constraints on the factors of the decompositions. In this paper, we propose an entirely different approac… ▽ More We consider the problem of identifying a parallel Wiener-Hammerstein structure from Volterra kernels. Methods based on Volterra kernels typically resort to coupled tensor decompositions of the kernels. However, in the case of parallel Wiener-Hammerstein systems, such methods require nontrivial constraints on the factors of the decompositions. In this paper, we propose an entirely different approach: by using special sampling (operating) points for the Jacobian of the nonlinear map from past inputs to the output, we can show that the Jacobian matrix becomes a linear projection of a tensor whose rank is equal to the number of branches. This representation allows us to solve the identification problem as a tensor recovery problem. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Report number: BioSiS

Journal ref: 19th IFAC Symposium on System Identification, SYSID 2021, Jul 2021, Padova (virtual), Italy

Determinantal Point Processes in the Flat Limit

Authors: Simon Barthelmé, Nicolas Tremblay, Konstantin Usevich, Pierre-Olivier Amblard

Abstract: Determinantal point processes (DPPs) are repulsive point processes where the interaction between points depends on the determinant of a positive-semi definite matrix. In this paper, we study the limiting process of L-ensembles based on kernel matrices, when the kernel function becomes flat (so that every point interacts with every other point, in a sense). We show that these limiting processes a… ▽ More Determinantal point processes (DPPs) are repulsive point processes where the interaction between points depends on the determinant of a positive-semi definite matrix. In this paper, we study the limiting process of L-ensembles based on kernel matrices, when the kernel function becomes flat (so that every point interacts with every other point, in a sense). We show that these limiting processes are best described in the formalism of extended L-ensembles and partial projection DPPs, and the exact limit depends mostly on the smoothness of the kernel function. In some cases, the limiting process is even universal, meaning that it does not depend on specifics of the kernel function, but only on its degree of smoothness. Since flat-limit DPPs are still repulsive processes, this implies that practically useful families of DPPs exist that do not require a spatial length-scale parameter. △ Less

Submitted 31 May, 2022; v1 submitted 15 July, 2021; originally announced July 2021.

Comments: Most of this material first appeared in arXiv:2007.04117, which has been split into two. The presentation has been simplified and some material is new

Extended L-ensembles: a new representation for Determinantal Point Processes

Authors: Nicolas Tremblay, Simon Barthelmé, Konstantin Usevich, Pierre-Olivier Amblard

Abstract: Determinantal point processes (DPPs) are a class of repulsive point processes, popular for their relative simplicity. They are traditionally defined via their marginal distributions, but a subset of DPPs called "L-ensembles" have tractable likelihoods and are thus particularly easy to work with. Indeed, in many applications, DPPs are more naturally defined based on the L-ensemble formulation rathe… ▽ More Determinantal point processes (DPPs) are a class of repulsive point processes, popular for their relative simplicity. They are traditionally defined via their marginal distributions, but a subset of DPPs called "L-ensembles" have tractable likelihoods and are thus particularly easy to work with. Indeed, in many applications, DPPs are more naturally defined based on the L-ensemble formulation rather than through the marginal kernel. The fact that not all DPPs are L-ensembles is unfortunate, but there is a unifying description. We introduce here extended L-ensembles, and show that all DPPs are extended L-ensembles (and vice-versa). Extended L-ensembles have very simple likelihood functions, contain L-ensembles and projection DPPs as special cases. From a theoretical standpoint, they fix some pathologies in the usual formalism of DPPs, for instance the fact that projection DPPs are not L-ensembles. From a practical standpoint, they extend the set of kernel functions that may be used to define DPPs: we show that conditional positive definite kernels are good candidates for defining DPPs, including DPPs that need no spatial scale parameter. Finally, extended L-ensembles are based on so-called ``saddle-point matrices'', and we prove an extension of the Cauchy-Binet theorem for such matrices that may be of independent interest. △ Less

Submitted 31 May, 2022; v1 submitted 12 July, 2021; originally announced July 2021.

Comments: Most of this material appeared in a previous arxiv submission (arXiv:2007.04117), two sections are new, and some things have been rephrased

Tensor-based framework for training flexible neural networks

Authors: Yassine Zniyed, Konstantin Usevich, Sebastian Miron, David Brie

Abstract: Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning al… ▽ More Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained canonical polyadic decomposition (CPD). The proposed algorithm can handle different decomposition bases. The goal of this method is to compress large pretrained NN models, by replacing subnetworks, {\em i.e.,} one or multiple layers of the original network, by a new flexible layer. The approach is applied to a pretrained convolutional neural network (CNN) used for character classification. △ Less

Submitted 25 June, 2021; originally announced June 2021.

Comments: 26 pages, 13 figures

MSC Class: 15A69; 68T07; 94A08; 12E05; 15A21

Convergence of gradient-based block coordinate descent algorithms for non-orthogonal joint approximate diagonalization of matrices

Authors: Jianze Li, Konstantin Usevich, Pierre Comon

Abstract: In this paper, we propose a gradient-based block coordinate descent (BCD-G) framework to solve the joint approximate diagonalization of matrices defined on the product of the complex Stiefel manifold and the special linear group. Instead of the cyclic fashion, we choose a block optimization based on the Riemannian gradient. To update the first block variable in the complex Stiefel manifold, we use… ▽ More In this paper, we propose a gradient-based block coordinate descent (BCD-G) framework to solve the joint approximate diagonalization of matrices defined on the product of the complex Stiefel manifold and the special linear group. Instead of the cyclic fashion, we choose a block optimization based on the Riemannian gradient. To update the first block variable in the complex Stiefel manifold, we use the well-known line search descent method. To update the second block variable in the special linear group, based on four kinds of different elementary transformations, we construct three classes: GLU, GQU and GU, and then get three BCD-G algorithms: BCD-GLU, BCD-GQU and BCD-GU. We establish the global and weak convergence of these three algorithms using the Łojasiewicz gradient inequality under the assumption that the iterates are bounded. We also propose a gradient-based Jacobi-type framework to solve the joint approximate diagonalization of matrices defined on the special linear group. As in the BCD-G case, using the GLU and GQU classes of elementary transformations, we focus on the Jacobi-GLU and Jacobi-GQU algorithms and establish their global and weak convergence. All the algorithms and convergence results described in this paper also apply to the real case. △ Less

Submitted 25 April, 2023; v1 submitted 28 September, 2020; originally announced September 2020.

Comments: 30 pages, 4 figures

MSC Class: 49M30; 65F99; 90C30; 15A23

Determinantal Point Processes in the Flat Limit: Extended L-ensembles, Partial-Projection DPPs and Universality Classes

Authors: Simon Barthelmé, Nicolas Tremblay, Konstantin Usevich, Pierre-Olivier Amblard

Abstract: Determinantal point processes (DPPs) are repulsive point processes where the interaction between points depends on the determinant of a positive-semi definite matrix. The contributions of this paper are two-fold. First of all, we introduce the concept of extended L-ensemble, a novel representation of DPPs. These extended L-ensembles are interesting objects because they fix some pathologies in the… ▽ More Determinantal point processes (DPPs) are repulsive point processes where the interaction between points depends on the determinant of a positive-semi definite matrix. The contributions of this paper are two-fold. First of all, we introduce the concept of extended L-ensemble, a novel representation of DPPs. These extended L-ensembles are interesting objects because they fix some pathologies in the usual formalism of DPPs, for instance the fact that projection DPPs are not L-ensembles. Every (fixed-size) DPP is an (fixed-size) extended L-ensemble, including projection DPPs. This new formalism enables to introduce and analyze a subclass of DPPs, called partial-projection DPPs. Secondly, with these new definitions in hand, we first show that partial-projection DPPs arise as perturbative limits of L-ensembles, that is, limits in $\varepsilon \rightarrow 0$ of L-ensembles based on matrices of the form $\varepsilon \mathbf{A} + \mathbf{B}$ where $\mathbf{B}$ is low-rank. We generalise this result by showing that partial-projection DPPs also arise as the limiting process of L-ensembles based on kernel matrices, when the kernel function becomes flat (so that every point interacts with every other point, in a sense). We show that the limiting point process depends mostly on the smoothness of the kernel function. In some cases, the limiting process is even universal, meaning that it does not depend on specifics of the kernel function, but only on its degree of smoothness. △ Less

Submitted 31 May, 2022; v1 submitted 8 July, 2020; originally announced July 2020.

Comments: This paper has now been divided in two parts, as explained in a paragraph before the abstract

MSC Class: 60G55

Coupled Tensor Decomposition for Hyperspectral and Multispectral Image Fusion with Inter-Image Variability

Authors: Ricardo Augusto Borsoi, Clémence Prévost, Konstantin Usevich, David Brie, José Carlos Moreira Bermudez, Cédric Richard

Abstract: Coupled tensor approximation has recently emerged as a promising approach for the fusion of hyperspectral and multispectral images, reconciling state of the art performance with strong theoretical guarantees. However, tensor-based approaches previously proposed assume that the different observed images are acquired under exactly the same conditions. A recent work proposed to accommodate inter-imag… ▽ More Coupled tensor approximation has recently emerged as a promising approach for the fusion of hyperspectral and multispectral images, reconciling state of the art performance with strong theoretical guarantees. However, tensor-based approaches previously proposed assume that the different observed images are acquired under exactly the same conditions. A recent work proposed to accommodate inter-image spectral variability in the image fusion problem using a matrix factorization-based formulation, but did not account for spatially-localized variations. Moreover, it lacks theoretical guarantees and has a high associated computational complexity. In this paper, we consider the image fusion problem while accounting for both spatially and spectrally localized changes in an additive model. We first study how the general identifiability of the model is impacted by the presence of such changes. Then, assuming that the high-resolution image and the variation factors admit a Tucker decomposition, two new algorithms are proposed -- one purely algebraic, and another based on an optimization procedure. Theoretical guarantees for the exact recovery of the high-resolution image are provided for both algorithms. Experimental results show that the proposed method outperforms state-of-the-art methods in the presence of spectral and spatial variations between the images, at a smaller computational cost. △ Less

Submitted 5 December, 2020; v1 submitted 30 June, 2020; originally announced June 2020.

On the convergence of Jacobi-type algorithms for Independent Component Analysis

Authors: Jianze Li, Konstantin Usevich, Pierre Comon

Abstract: Jacobi-type algorithms for simultaneous approximate diagonalization of real (or complex) symmetric tensors have been widely used in independent component analysis (ICA) because of their good performance. One natural way of choosing the index pairs in Jacobi-type algorithms is the classical cyclic ordering, while the other way is based on the Riemannian gradient in each iteration. In this paper, we… ▽ More Jacobi-type algorithms for simultaneous approximate diagonalization of real (or complex) symmetric tensors have been widely used in independent component analysis (ICA) because of their good performance. One natural way of choosing the index pairs in Jacobi-type algorithms is the classical cyclic ordering, while the other way is based on the Riemannian gradient in each iteration. In this paper, we mainly review in an accessible manner our recent results in a series of papers about weak and global convergence of these Jacobi-type algorithms. These results are mainly based on the Lojasiewicz gradient inequality. △ Less

Submitted 15 June, 2020; v1 submitted 16 December, 2019; originally announced December 2019.

Comments: 5 pages

Journal ref: IEEE SAM 2020

Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis

Authors: Jianze Li, Konstantin Usevich, Pierre Comon

Abstract: In this paper, we propose a Jacobi-type algorithm to solve the low rank orthogonal approximation problem of symmetric tensors. This algorithm includes as a special case the well-known Jacobi CoM2 algorithm for the approximate orthogonal diagonalization problem of symmetric tensors. We first prove the weak convergence of this algorithm, \textit{i.e.} any accumulation point is a stationary point. Th… ▽ More In this paper, we propose a Jacobi-type algorithm to solve the low rank orthogonal approximation problem of symmetric tensors. This algorithm includes as a special case the well-known Jacobi CoM2 algorithm for the approximate orthogonal diagonalization problem of symmetric tensors. We first prove the weak convergence of this algorithm, \textit{i.e.} any accumulation point is a stationary point. Then we study the global convergence of this algorithm under a gradient based ordering for a special case: the best rank-2 orthogonal approximation of 3rd order symmetric tensors, and prove that an accumulation point is the unique limit point under some conditions. Numerical experiments are presented to show the efficiency of this algorithm. △ Less

Submitted 30 March, 2021; v1 submitted 2 November, 2019; originally announced November 2019.

Comments: 19 pages, 4 figures

MSC Class: 15A69; 15A23; 49M30; 65F99; 26E05

Spectral properties of kernel matrices in the flat limit

Authors: Simon Barthelmé, Konstantin Usevich

Abstract: Kernel matrices are of central importance to many applied fields. In this manuscript, we focus on spectral properties of kernel matrices in the so-called ``flat limit'', which occurs when points are close together relative to the scale of the kernel. We establish asymptotic expressions for the determinants of the kernel matrices, which we then leverage to obtain asymptotic expressions for the main… ▽ More Kernel matrices are of central importance to many applied fields. In this manuscript, we focus on spectral properties of kernel matrices in the so-called ``flat limit'', which occurs when points are close together relative to the scale of the kernel. We establish asymptotic expressions for the determinants of the kernel matrices, which we then leverage to obtain asymptotic expressions for the main terms of the eigenvalues. Analyticity of the eigenprojectors yields expressions for limiting eigenvectors, which are strongly tied to discrete orthogonal polynomials. Both smooth and finitely smooth kernels are covered, with stronger results available in the finite smoothness case. △ Less

Submitted 27 March, 2025; v1 submitted 30 October, 2019; originally announced October 2019.

Comments: 41 pages, 8 pictures

MSC Class: 15A18; 47A55; 47A75; 47B34; 60G15; 65D05

Journal ref: Siam J. Matrix Anal. Appl., 42(1):17-57, 2021

Approximate matrix and tensor diagonalization by unitary transformations: convergence of Jacobi-type algorithms

Authors: Konstantin Usevich, Jianze Li, Pierre Comon

Abstract: We propose a gradient-based Jacobi algorithm for a class of maximization problems on the unitary group, with a focus on approximate diagonalization of complex matrices and tensors by unitary transformations. We provide weak convergence results, and prove local linear convergence of this algorithm.The convergence results also apply to the case of real-valued tensors. We propose a gradient-based Jacobi algorithm for a class of maximization problems on the unitary group, with a focus on approximate diagonalization of complex matrices and tensors by unitary transformations. We provide weak convergence results, and prove local linear convergence of this algorithm.The convergence results also apply to the case of real-valued tensors. △ Less

Submitted 10 July, 2020; v1 submitted 29 May, 2019; originally announced May 2019.

Hyperspectral Super-Resolution with Coupled Tucker Approximation: Recoverability and SVD-based algorithms

Authors: Clémence Prévost, Konstantin Usevich, Pierre Comon, David Brie

Abstract: We propose a novel approach for hyperspectral super-resolution, that is based on low-rank tensor approximation for a coupled low-rank multilinear (Tucker) model. We show that the correct recovery holds for a wide range of multilinear ranks. For coupled tensor approximation, we propose two SVD-based algorithms that are simple and fast, but with a performance comparable to the state-of-the-art… ▽ More We propose a novel approach for hyperspectral super-resolution, that is based on low-rank tensor approximation for a coupled low-rank multilinear (Tucker) model. We show that the correct recovery holds for a wide range of multilinear ranks. For coupled tensor approximation, we propose two SVD-based algorithms that are simple and fast, but with a performance comparable to the state-of-the-art methods. The approach is applicable to the case of unknown spatial degradation and to the pansharpening problem. △ Less

Submitted 20 January, 2020; v1 submitted 21 November, 2018; originally announced November 2018.

Comments: IEEE Transactions on Signal Processing, Institute of Electrical and Electronics Engineers, in Press

On approximate diagonalization of third order symmetric tensors by orthogonal transformations

Authors: Jianze Li, Konstantin Usevich, Pierre Comon

Abstract: In this paper, we study the approximate orthogonal diagonalization problem of third order symmetric tensors. We define several classes of approximately diagonal tensors, including the ones corresponding to the stationary points of this problem. We study the relationships between these classes, and other well-known objects, such as tensor Z-eigenvalue and Z-eigenvector. We also prove results on con… ▽ More In this paper, we study the approximate orthogonal diagonalization problem of third order symmetric tensors. We define several classes of approximately diagonal tensors, including the ones corresponding to the stationary points of this problem. We study the relationships between these classes, and other well-known objects, such as tensor Z-eigenvalue and Z-eigenvector. We also prove results on convergence of the cyclic Jacobi (or Jacobi CoM2) algorithm. △ Less

Submitted 4 April, 2019; v1 submitted 4 April, 2018; originally announced April 2018.

Comments: 24 pages

MSC Class: 15A69; 65F99; 90C30

Structured low-rank matrix completion for forecasting in time series analysis

Authors: Jonathan Gillard, Konstantin Usevich

Abstract: In this paper we consider the low-rank matrix completion problem with specific application to forecasting in time series analysis. Briefly, the low-rank matrix completion problem is the problem of imputing missing values of a matrix under a rank constraint. We consider a matrix completion problem for Hankel matrices and a convex relaxation based on the nuclear norm. Based on new theoretical result… ▽ More In this paper we consider the low-rank matrix completion problem with specific application to forecasting in time series analysis. Briefly, the low-rank matrix completion problem is the problem of imputing missing values of a matrix under a rank constraint. We consider a matrix completion problem for Hankel matrices and a convex relaxation based on the nuclear norm. Based on new theoretical results and a number of numerical and real examples, we investigate the cases when the proposed approach can work. Our results highlight the importance of choosing a proper weighting scheme for the known observations. △ Less

Submitted 22 February, 2018; originally announced February 2018.

Comments: 25 pages, 12 figures

Decoupling multivariate polynomials: interconnections between tensorizations

Authors: Konstantin Usevich, Philippe Dreesen, Mariya Ishteva

Abstract: Decoupling multivariate polynomials is useful for obtaining an insight into the workings of a nonlinear mapping, performing parameter reduction, or approximating nonlinear functions. Several different tensor-based approaches have been proposed independently for this task, involving different tensor representations of the functions, and ultimately leading to a canonical polyadic decomposition. We… ▽ More Decoupling multivariate polynomials is useful for obtaining an insight into the workings of a nonlinear mapping, performing parameter reduction, or approximating nonlinear functions. Several different tensor-based approaches have been proposed independently for this task, involving different tensor representations of the functions, and ultimately leading to a canonical polyadic decomposition. We first show that the involved tensors are related by a linear transformation, and that their CP decompositions and uniqueness properties are closely related. This connection provides a way to better assess which of the methods should be favored in certain problem settings, and may be a starting point to unify the two approaches. Second, we show that taking into account the previously ignored intrinsic structure in the tensor decompositions improves the uniqueness properties of the decompositions and thus enlarges the applicability range of the methods. △ Less

Submitted 29 January, 2019; v1 submitted 7 March, 2017; originally announced March 2017.

Comments: 20 pages, 2 figures

MSC Class: 12E05; 15A21; 15A69

Globally convergent Jacobi-type algorithms for simultaneous orthogonal symmetric tensor diagonalization

Authors: Jianze Li, Konstantin Usevich, Pierre Comon

Abstract: In this paper, we consider a family of Jacobi-type algorithms for simultaneous orthogonal diagonalization problem of symmetric tensors. For the Jacobi-based algorithm of [SIAM J. Matrix Anal. Appl., 2(34):651--672, 2013], we prove its global convergence for simultaneous orthogonal diagonalization of symmetric matrices and 3rd-order tensors. We also propose a new Jacobi-based algorithm in the gener… ▽ More In this paper, we consider a family of Jacobi-type algorithms for simultaneous orthogonal diagonalization problem of symmetric tensors. For the Jacobi-based algorithm of [SIAM J. Matrix Anal. Appl., 2(34):651--672, 2013], we prove its global convergence for simultaneous orthogonal diagonalization of symmetric matrices and 3rd-order tensors. We also propose a new Jacobi-based algorithm in the general setting and prove its global convergence for sufficiently smooth functions. △ Less

Submitted 27 July, 2017; v1 submitted 13 February, 2017; originally announced February 2017.

Comments: 22 pages, 6 figures

MSC Class: 15A69; 49M30; 65F99; 90C30

Identifiability of an X-rank decomposition of polynomial maps

Authors: Pierre Comon, Yang Qi, Konstantin Usevich

Abstract: In this paper, we study a polynomial decomposition model that arises in problems of system identification, signal processing and machine learning. We show that this decomposition is a special case of the X-rank decomposition --- a powerful novel concept in algebraic geometry that generalizes the tensor CP decomposition. We prove new results on generic/maximal rank and on identifiability of a parti… ▽ More In this paper, we study a polynomial decomposition model that arises in problems of system identification, signal processing and machine learning. We show that this decomposition is a special case of the X-rank decomposition --- a powerful novel concept in algebraic geometry that generalizes the tensor CP decomposition. We prove new results on generic/maximal rank and on identifiability of a particular polynomial decomposition model. In the paper, we try to make results and basic tools accessible for general audience (assuming no knowledge of algebraic geometry or its prerequisites). △ Less

Submitted 5 April, 2017; v1 submitted 4 March, 2016; originally announced March 2016.

Comments: 26 pages

MSC Class: 12E05; 14M12; 15A21; 15A69

Quasi-Hankel low-rank matrix completion: a convex relaxation

Authors: Konstantin Usevich, Pierre Comon

Abstract: The completion of matrices with missing values under the rank constraint is a non-convex optimization problem. A popular convex relaxation is based on minimization of the nuclear norm (sum of singular values) of the matrix. For this relaxation, an important question is whether the two optimization problems lead to the same solution. This question was addressed in the literature mostly in the case… ▽ More The completion of matrices with missing values under the rank constraint is a non-convex optimization problem. A popular convex relaxation is based on minimization of the nuclear norm (sum of singular values) of the matrix. For this relaxation, an important question is whether the two optimization problems lead to the same solution. This question was addressed in the literature mostly in the case of random positions of missing elements and random known elements. In this contribution, we analyze the case of structured matrices with fixed pattern of missing values, in particular, the case of Hankel and quasi-Hankel matrix completion, which appears as a subproblem in the computation of symmetric tensor canonical polyadic decomposition. We extend existing results on completion of rank-one real Hankel matrices to completion of rank-r complex Hankel and quasi-Hankel matrices. △ Less

Submitted 9 June, 2015; v1 submitted 28 May, 2015; originally announced May 2015.

Comments: 28 pages, 5 figures

Adjusted least squares fitting of algebraic hypersurfaces

Authors: Konstantin Usevich, Ivan Markovsky

Abstract: We consider the problem of fitting a set of points in Euclidean space by an algebraic hypersurface. We assume that points on a true hypersurface, described by a polynomial equation, are corrupted by zero mean independent Gaussian noise, and we estimate the coefficients of the true polynomial equation. The adjusted least squares estimator accounts for the bias present in the ordinary least squares… ▽ More We consider the problem of fitting a set of points in Euclidean space by an algebraic hypersurface. We assume that points on a true hypersurface, described by a polynomial equation, are corrupted by zero mean independent Gaussian noise, and we estimate the coefficients of the true polynomial equation. The adjusted least squares estimator accounts for the bias present in the ordinary least squares estimator. The adjusted least squares estimator is based on constructing a quasi-Hankel matrix, which is a bias-corrected matrix of moments. For the case of unknown noise variance, the estimator is defined as a solution of a polynomial eigenvalue problem. In this paper, we present new results on invariance properties of the adjusted least squares estimator and an improved algorithm for computing the estimator for an arbitrary set of monomials in the polynomial equation. △ Less

Submitted 20 August, 2015; v1 submitted 6 December, 2014; originally announced December 2014.

Comments: 30 pages, 10 figures

MSC Class: 15A22; 15B05; 33C45; 62H12; 65D10; 65F15; 68U05

On certain multivariate Vandermonde determinants whose variables separate

Authors: Stefano De Marchi, Konstantin Usevich

Abstract: We prove that for almost square tensor product grids and certain sets of bivariate polynomials the Vandermonde determinant can be factored into a product of univariate Vandermonde determinants. This result generalizes the conjecture [Lemma 1, L. Bos et al. (2009), Dolomites Research Notes on Approximation, 2:1-15]. As a special case, we apply the result to Padua and Padua-like points. We prove that for almost square tensor product grids and certain sets of bivariate polynomials the Vandermonde determinant can be factored into a product of univariate Vandermonde determinants. This result generalizes the conjecture [Lemma 1, L. Bos et al. (2009), Dolomites Research Notes on Approximation, 2:1-15]. As a special case, we apply the result to Padua and Padua-like points. △ Less

Submitted 11 March, 2014; v1 submitted 25 November, 2013; originally announced November 2013.

Comments: 10 pages, 1 figure

MSC Class: 15A15; 15B99; 41A05

Journal ref: Linear Algebra and its Applications, Volume 449, Pages 17--27, 2014

Multivariate and 2D Extensions of Singular Spectrum Analysis with the Rssa Package

Authors: Nina Golyandina, Anton Korobeynikov, Alex Shlemov, Konstantin Usevich

Abstract: Implementation of multivariate and 2D extensions of Singular Spectrum Analysis (SSA) by means of the R-package Rssa is considered. The extensions include MSSA for simultaneous analysis and forecasting of several time series and 2D-SSA for analysis of digital images. A new extension of 2D-SSA analysis called Shaped 2D-SSA is introduced for analysis of images of arbitrary shape, not necessary rectan… ▽ More Implementation of multivariate and 2D extensions of Singular Spectrum Analysis (SSA) by means of the R-package Rssa is considered. The extensions include MSSA for simultaneous analysis and forecasting of several time series and 2D-SSA for analysis of digital images. A new extension of 2D-SSA analysis called Shaped 2D-SSA is introduced for analysis of images of arbitrary shape, not necessary rectangular. It is shown that implementation of Shaped 2D-SSA can serve as a base for implementation of MSSA and other generalizations. Efficient implementation of operations with Hankel and Hankel-block-Hankel matrices through FFT is suggested. Examples with code fragments in R, which explain the methodology and demonstrate the proper use of Rssa, are presented. △ Less

Submitted 19 September, 2014; v1 submitted 19 September, 2013; originally announced September 2013.

Journal ref: Journal of Statistical Software, v.67, Issue 2, 2015, p. 1-78

Factorization approach to structured low-rank approximation with applications

Authors: Mariya Ishteva, Konstantin Usevich, Ivan Markovsky

Abstract: We consider the problem of approximating an affinely structured matrix, for example a Hankel matrix, by a low-rank matrix with the same structure. This problem occurs in system identification, signal processing and computer algebra, among others. We impose the low-rank by modeling the approximation as a product of two factors with reduced dimension. The structure of the low-rank model is enforced… ▽ More We consider the problem of approximating an affinely structured matrix, for example a Hankel matrix, by a low-rank matrix with the same structure. This problem occurs in system identification, signal processing and computer algebra, among others. We impose the low-rank by modeling the approximation as a product of two factors with reduced dimension. The structure of the low-rank model is enforced by introducing a penalty term in the objective function. The proposed local optimization algorithm is able to solve the weighted structured low-rank approximation problem, as well as to deal with the cases of missing or fixed elements. In contrast to approaches based on kernel representations (in linear algebraic sense), the proposed algorithm is designed to address the case of small targeted rank. We compare it to existing approaches on numerical examples of system identification, approximate greatest common divisor problem, and symmetric tensor decomposition and demonstrate its consistently good performance. △ Less

Submitted 24 June, 2014; v1 submitted 8 August, 2013; originally announced August 2013.

Comments: Accepted for publication in SIAM Journal on Matrix Analysis and Applications (SIMAX)

MSC Class: 15A23; 15A83; 65F99; 93B30; 37M10; 37N30; 11A05; 15A69

Variable projection methods for approximate (greatest) common divisor computations

Authors: Konstantin Usevich, Ivan Markovsky

Abstract: We consider the problem of finding for a given $N$-tuple of polynomials (real or complex) the closest $N$-tuple that has a common divisor of degree at least $d$. Extended weighted Euclidean seminorm of the coefficients is used as a measure of closeness. Two equivalent representations of the problem are considered: (i) direct parameterization over the common divisors and quotients (image representa… ▽ More We consider the problem of finding for a given $N$-tuple of polynomials (real or complex) the closest $N$-tuple that has a common divisor of degree at least $d$. Extended weighted Euclidean seminorm of the coefficients is used as a measure of closeness. Two equivalent representations of the problem are considered: (i) direct parameterization over the common divisors and quotients (image representation), and (ii) Sylvester low-rank approximation (kernel representation). We use the duality between least-squares and least-norm problems to show that (i) and (ii) are closely related to mosaic Hankel low-rank approximation. This allows us to apply to the approximate common divisor problem recent results on complexity and accuracy of computations for mosaic Hankel low-rank approximation. We develop optimization methods based on the variable projection principle both for image and kernel representation. These methods have linear complexity in the degrees of the polynomials for small and large $d$. We provide a software implementation of the developed methods, which is based on a software package for structured low-rank approximation. △ Less

Submitted 4 November, 2015; v1 submitted 25 April, 2013; originally announced April 2013.

Comments: 32 pages, 4 figures

MSC Class: 15B05; 15B99; 41A29; 65K05; 65Y20; 68W25

Variable projection for affinely structured low-rank approximation in weighted 2-norms

Authors: Konstantin Usevich, Ivan Markovsky

Abstract: The structured low-rank approximation problem for general affine structures, weighted 2-norms and fixed elements is considered. The variable projection principle is used to reduce the dimensionality of the optimization problem. Algorithms for evaluation of the cost function, the gradient and an approximation of the Hessian are developed. For $m \times n$ mosaic Hankel matrices the algorithms have… ▽ More The structured low-rank approximation problem for general affine structures, weighted 2-norms and fixed elements is considered. The variable projection principle is used to reduce the dimensionality of the optimization problem. Algorithms for evaluation of the cost function, the gradient and an approximation of the Hessian are developed. For $m \times n$ mosaic Hankel matrices the algorithms have complexity $O(m^2 n)$. △ Less

Submitted 25 July, 2013; v1 submitted 16 November, 2012; originally announced November 2012.

Comments: 25 pages, 4 figures

MSC Class: 15B99; 15B05; 41A29; 49M30; 65F30; 65K05; 65Y20

On signal and extraneous roots in Singular Spectrum Analysis

Authors: Konstantin Usevich

Abstract: In the present paper we study properties of roots of characteristic polynomials for the linear recurrent formulae (LRF) that govern time series. We also investigate how the values of these roots affect Singular Spectrum Analysis implications, in what concerns separation of components, SSA forecasting and related signal parameter estimation methods. The roots of the characteristic polynomial for an… ▽ More In the present paper we study properties of roots of characteristic polynomials for the linear recurrent formulae (LRF) that govern time series. We also investigate how the values of these roots affect Singular Spectrum Analysis implications, in what concerns separation of components, SSA forecasting and related signal parameter estimation methods. The roots of the characteristic polynomial for an LRF comprise the signal roots, which determine the structure of the time series, and extraneous roots. We show how the separability of two time series can be characterized in terms of their signal roots. All possible cases of exact separability are enumerated. We also examine properties of extraneous roots of the LRF used in SSA forecasting algorithms, which is equivalent to the Min-Norm vector in subspace-based estimation methods. We apply recent theoretical results for orthogonal polynomials on the unit circle, which enable us to precisely describe the asymptotic distribution of extraneous roots relative to the position of the signal roots. △ Less

Submitted 17 June, 2010; originally announced June 2010.

Comments: 24 pages, 7 figures

Journal ref: Statistics and Its Interface, 2010, Vol. 3(3):281-295