Showing 1–18 of 18 results for author: Király, F

Probabilistic supervised learning

Authors: Frithjof Gressmann, Franz J. Király, Bilal Mateen, Harald Oberhauser

Abstract: Predictive modelling and supervised learning are central to modern data science. With predictions from an ever-expanding number of supervised black-box strategies - e.g., kernel methods, random forests, deep learning aka neural networks - being employed as a basis for decision making processes, it is crucial to understand the statistical uncertainty associated with these predictions. As a genera… ▽ More Predictive modelling and supervised learning are central to modern data science. With predictions from an ever-expanding number of supervised black-box strategies - e.g., kernel methods, random forests, deep learning aka neural networks - being employed as a basis for decision making processes, it is crucial to understand the statistical uncertainty associated with these predictions. As a general means to approach the issue, we present an overarching framework for black-box prediction strategies that not only predict the target but also their own predictions' uncertainty. Moreover, the framework allows for fair assessment and comparison of disparate prediction strategies. For this, we formally consider strategies capable of predicting full distributions from feature variables, so-called probabilistic supervised learning strategies. Our work draws from prior work including Bayesian statistics, information theory, and modern supervised machine learning, and in a novel synthesis leads to (a) new theoretical insights such as a probabilistic bias-variance decomposition and an entropic formulation of prediction, as well as to (b) new algorithms and meta-algorithms, such as composite prediction strategies, probabilistic boosting and bagging, and a probabilistic predictive independence test. Our black-box formulation also leads (c) to a new modular interface view on probabilistic supervised learning and a modelling workflow API design, which we have implemented in the newly released skpro machine learning toolbox, extending the familiar modelling interface and meta-modelling functionality of sklearn. The skpro package provides interfaces for construction, composition, and tuning of probabilistic supervised learning strategies, together with orchestration features for validation and comparison of any such strategy - be it frequentist, Bayesian, or other. △ Less

Submitted 7 May, 2019; v1 submitted 2 January, 2018; originally announced January 2018.

Predictive Independence Testing, Predictive Conditional Independence Testing, and Predictive Graphical Modelling

Authors: Samuel Burkart, Franz J Király

Abstract: Testing (conditional) independence of multivariate random variables is a task central to statistical inference and modelling in general - though unfortunately one for which to date there does not exist a practicable workflow. State-of-art workflows suffer from the need for heuristic or subjective manual choices, high computational complexity, or strong parametric assumptions. We address these pr… ▽ More Testing (conditional) independence of multivariate random variables is a task central to statistical inference and modelling in general - though unfortunately one for which to date there does not exist a practicable workflow. State-of-art workflows suffer from the need for heuristic or subjective manual choices, high computational complexity, or strong parametric assumptions. We address these problems by establishing a theoretical link between multivariate/conditional independence testing, and model comparison in the multivariate predictive modelling aka supervised learning task. This link allows advances in the extensively studied supervised learning workflow to be directly transferred to independence testing workflows - including automated tuning of machine learning type which addresses the need for a heuristic choice, the ability to quantitatively trade-off computational demand with accuracy, and the modern black-box philosophy for checking and interfacing. As a practical implementation of this link between the two workflows, we present a python package 'pcit', which implements our novel multivariate and conditional independence tests, interfacing the supervised learning API of the scikit-learn package. Theory and package also allow for straightforward independence test based learning of graphical model structure. We empirically show that our proposed predictive independence test outperform or are on par to current practice, and the derived graphical model structure learning algorithms asymptotically recover the 'true' graph. This paper, and the 'pcit' package accompanying it, thus provide powerful, scalable, generalizable, and easy-to-use methods for multivariate and conditional independence testing, as well as for graphical model structure learning. △ Less

Submitted 28 April, 2018; v1 submitted 15 November, 2017; originally announced November 2017.

Kernels for sequentially ordered data

Authors: Franz J Király, Harald Oberhauser

Abstract: We present a novel framework for kernel learning with sequential data of any kind, such as time series, sequences of graphs, or strings. Our approach is based on signature features which can be seen as an ordered variant of sample (cross-)moments; it allows to obtain a "sequentialized" version of any static kernel. The sequential kernels are efficiently computable for discrete sequences and are sh… ▽ More We present a novel framework for kernel learning with sequential data of any kind, such as time series, sequences of graphs, or strings. Our approach is based on signature features which can be seen as an ordered variant of sample (cross-)moments; it allows to obtain a "sequentialized" version of any static kernel. The sequential kernels are efficiently computable for discrete sequences and are shown to approximate a continuous moment form in a sampling sense. A number of known kernels for sequences arise as "sequentializations" of suitable static kernels: string kernels may be obtained as a special case, and alignment kernels are closely related up to a modification that resolves their open non-definiteness issue. Our experiments indicate that our signature-based sequential kernel framework may be a promising approach to learning with sequential data, such as time series, that allows to avoid extensive manual pre-processing. △ Less

Submitted 29 January, 2016; originally announced January 2016.

Phase retrieval using random cubatures and fusion frames of positive semidefinite matrices

Authors: Martin Ehler, Manuel Graef, Franz J. Kiraly

Abstract: As a generalization of the standard phase retrieval problem, we seek to reconstruct symmetric rank-1 matrices from inner products with subclasses of positive semidefinite matrices. For such subclasses, we introduce random cubatures for spaces of multivariate polynomials based on moment conditions. The inner products with samples from sufficiently strong random cubatures allow the reconstruction of… ▽ More As a generalization of the standard phase retrieval problem, we seek to reconstruct symmetric rank-1 matrices from inner products with subclasses of positive semidefinite matrices. For such subclasses, we introduce random cubatures for spaces of multivariate polynomials based on moment conditions. The inner products with samples from sufficiently strong random cubatures allow the reconstruction of symmetric rank-1 matrices with a decent probability by solving the feasibility problem of a semidefinite program. △ Less

Submitted 19 May, 2015; originally announced May 2015.

Learning with Algebraic Invariances, and the Invariant Kernel Trick

Authors: Franz J. Király, Andreas Ziehe, Klaus-Robert Müller

Abstract: When solving data analysis problems it is important to integrate prior knowledge and/or structural invariances. This paper contributes by a novel framework for incorporating algebraic invariance structure into kernels. In particular, we show that algebraic properties such as sign symmetries in data, phase independence, scaling etc. can be included easily by essentially performing the kernel trick… ▽ More When solving data analysis problems it is important to integrate prior knowledge and/or structural invariances. This paper contributes by a novel framework for incorporating algebraic invariance structure into kernels. In particular, we show that algebraic properties such as sign symmetries in data, phase independence, scaling etc. can be included easily by essentially performing the kernel trick twice. We demonstrate the usefulness of our theory in simulations on selected applications such as sign-invariant spectral clustering and underdetermined ICA. △ Less

Submitted 28 November, 2014; originally announced November 2014.

Learning with Cross-Kernels and Ideal PCA

Authors: Franz J Király, Martin Kreuzer, Louis Theran

Abstract: We describe how cross-kernel matrices, that is, kernel matrices between the data and a custom chosen set of `feature spanning points' can be used for learning. The main potential of cross-kernels lies in the fact that (a) only one side of the matrix scales with the number of data points, and (b) cross-kernels, as opposed to the usual kernel matrices, can be used to certify for the data manifold. O… ▽ More We describe how cross-kernel matrices, that is, kernel matrices between the data and a custom chosen set of `feature spanning points' can be used for learning. The main potential of cross-kernels lies in the fact that (a) only one side of the matrix scales with the number of data points, and (b) cross-kernels, as opposed to the usual kernel matrices, can be used to certify for the data manifold. Our theoretical framework, which is based on a duality involving the feature space and vanishing ideals, indicates that cross-kernels have the potential to be used for any kind of kernel learning. We present a novel algorithm, Ideal PCA (IPCA), which cross-kernelizes PCA. We demonstrate on real and synthetic data that IPCA allows to (a) obtain PCA-like features faster and (b) to extract novel and empirically validated features certifying for the data manifold. △ Less

Submitted 10 June, 2014; originally announced June 2014.

Matroid Regression

Authors: Franz J Király, Louis Theran

Abstract: We propose an algebraic combinatorial method for solving large sparse linear systems of equations locally - that is, a method which can compute single evaluations of the signal without computing the whole signal. The method scales only in the sparsity of the system and not in its size, and allows to provide error estimates for any solution method. At the heart of our approach is the so-called regr… ▽ More We propose an algebraic combinatorial method for solving large sparse linear systems of equations locally - that is, a method which can compute single evaluations of the signal without computing the whole signal. The method scales only in the sparsity of the system and not in its size, and allows to provide error estimates for any solution method. At the heart of our approach is the so-called regression matroid, a combinatorial object associated to sparsity patterns, which allows to replace inversion of the large matrix with the inversion of a kernel matrix that is constant size. We show that our method provides the best linear unbiased estimator (BLUE) for this setting and the minimum variance unbiased estimator (MVUE) under Gaussian noise assumptions, and furthermore we show that the size of the kernel matrix which is to be inverted can be traded off with accuracy. △ Less

Submitted 4 March, 2014; originally announced March 2014.

The Algebraic Approach to Phase Retrieval and Explicit Inversion at the Identifiability Threshold

Authors: Franz J Király, Martin Ehler

Abstract: We study phase retrieval from magnitude measurements of an unknown signal as an algebraic estimation problem. Indeed, phase retrieval from rank-one and more general linear measurements can be treated in an algebraic way. It is verified that a certain number of generic rank-one or generic linear measurements are sufficient to enable signal reconstruction for generic signals, and slightly more gener… ▽ More We study phase retrieval from magnitude measurements of an unknown signal as an algebraic estimation problem. Indeed, phase retrieval from rank-one and more general linear measurements can be treated in an algebraic way. It is verified that a certain number of generic rank-one or generic linear measurements are sufficient to enable signal reconstruction for generic signals, and slightly more generic measurements yield reconstructability for all signals. Our results solve a few open problems stated in the recent literature. Furthermore, we show how the algebraic estimation problem can be solved by a closed-form algebraic estimation technique, termed ideal regression, providing non-asymptotic success guarantees. △ Less

Submitted 17 February, 2014; originally announced February 2014.

Dual-to-kernel learning with ideals

Authors: Franz J. Király, Martin Kreuzer, Louis Theran

Abstract: In this paper, we propose a theory which unifies kernel learning and symbolic algebraic methods. We show that both worlds are inherently dual to each other, and we use this duality to combine the structure-awareness of algebraic methods with the efficiency and generality of kernels. The main idea lies in relating polynomial rings to feature space, and ideals to manifolds, then exploiting this gene… ▽ More In this paper, we propose a theory which unifies kernel learning and symbolic algebraic methods. We show that both worlds are inherently dual to each other, and we use this duality to combine the structure-awareness of algebraic methods with the efficiency and generality of kernels. The main idea lies in relating polynomial rings to feature space, and ideals to manifolds, then exploiting this generative-discriminative duality on kernel matrices. We illustrate this by proposing two algorithms, IPCA and AVICA, for simultaneous manifold and feature learning, and test their accuracy on synthetic and real world data. △ Less

Submitted 1 February, 2014; originally announced February 2014.

Comments: 15 pages, 1 figure

Algebraic matroids with graph symmetry

Authors: Franz J. Király, Zvi Rosen, Louis Theran

Abstract: This paper studies the properties of two kinds of matroids: (a) algebraic matroids and (b) finite and infinite matroids whose ground set have some canonical symmetry, for example row and column symmetry and transposition symmetry. For (a) algebraic matroids, we expose cryptomorphisms making them accessible to techniques from commutative algebra. This allows us to introduce for each circuit in an… ▽ More This paper studies the properties of two kinds of matroids: (a) algebraic matroids and (b) finite and infinite matroids whose ground set have some canonical symmetry, for example row and column symmetry and transposition symmetry. For (a) algebraic matroids, we expose cryptomorphisms making them accessible to techniques from commutative algebra. This allows us to introduce for each circuit in an algebraic matroid an invariant called circuit polynomial, generalizing the minimal poly- nomial in classical Galois theory, and studying the matroid structure with multivariate methods. For (b) matroids with symmetries we introduce combinatorial invariants capturing structural properties of the rank function and its limit behavior, and obtain proofs which are purely combinatorial and do not assume algebraicity of the matroid; these imply and generalize known results in some specific cases where the matroid is also algebraic. These results are motivated by, and readily applicable to framework rigidity, low-rank matrix completion and determinantal varieties, which lie in the intersection of (a) and (b) where additional results can be derived. We study the corresponding matroids and their associated invariants, and for selected cases, we characterize the matroidal structure and the circuit polynomials completely. △ Less

Submitted 13 December, 2013; originally announced December 2013.

Efficient Orthogonal Tensor Decomposition, with an Application to Latent Variable Model Learning

Authors: Franz J. Király

Abstract: Decomposing tensors into orthogonal factors is a well-known task in statistics, machine learning, and signal processing. We study orthogonal outer product decompositions where the factors in the summands in the decomposition are required to be orthogonal across summands, by relating this orthogonal decomposition to the singular value decompositions of the flattenings. We show that it is a non-triv… ▽ More Decomposing tensors into orthogonal factors is a well-known task in statistics, machine learning, and signal processing. We study orthogonal outer product decompositions where the factors in the summands in the decomposition are required to be orthogonal across summands, by relating this orthogonal decomposition to the singular value decompositions of the flattenings. We show that it is a non-trivial assumption for a tensor to have such an orthogonal decomposition, and we show that it is unique (up to natural symmetries) in case it exists, in which case we also demonstrate how it can be efficiently and reliably obtained by a sequence of singular value decompositions. We demonstrate how the factoring algorithm can be applied for parameter identification in latent variable and mixture models. △ Less

Submitted 12 September, 2013; originally announced September 2013.

Obtaining error-minimizing estimates and universal entry-wise error bounds for low-rank matrix completion

Authors: Franz J. Király, Louis Theran

Abstract: We propose a general framework for reconstructing and denoising single entries of incomplete and noisy entries. We describe: effective algorithms for deciding if and entry can be reconstructed and, if so, for reconstructing and denoising it; and a priori bounds on the error of each entry, individually. In the noiseless case our algorithm is exact. For rank-one matrices, the new algorithm is fast,… ▽ More We propose a general framework for reconstructing and denoising single entries of incomplete and noisy entries. We describe: effective algorithms for deciding if and entry can be reconstructed and, if so, for reconstructing and denoising it; and a priori bounds on the error of each entry, individually. In the noiseless case our algorithm is exact. For rank-one matrices, the new algorithm is fast, admits a highly-parallel implementation, and produces an error minimizing estimate that is qualitatively close to our theoretical and the state-of-the-are Nuclear Norm and OptSpace methods. △ Less

Submitted 29 March, 2013; v1 submitted 21 February, 2013; originally announced February 2013.

Comments: 14 pages with appendix, 2 figures, v2 adds larger experiments and smoother exposition

Coherence and sufficient sampling densities for reconstruction in compressed sensing

Authors: Franz J. Király, Louis Theran

Abstract: We give a new, very general, formulation of the compressed sensing problem in terms of coordinate projections of an analytic variety, and derive sufficient sampling rates for signal reconstruction. Our bounds are linear in the coherence of the signal space, a geometric parameter independent of the specific signal and measurement, and logarithmic in the ambient dimension where the signal is present… ▽ More We give a new, very general, formulation of the compressed sensing problem in terms of coordinate projections of an analytic variety, and derive sufficient sampling rates for signal reconstruction. Our bounds are linear in the coherence of the signal space, a geometric parameter independent of the specific signal and measurement, and logarithmic in the ambient dimension where the signal is presented. We exemplify our approach by deriving sufficient sampling densities for low-rank matrix completion and distance matrix completion which are independent of the true matrix. △ Less

Submitted 2 November, 2013; v1 submitted 12 February, 2013; originally announced February 2013.

Comments: 16 pages, 1 figure. v2 streamlines the exposition

Fano schemes of generic intersections and machine learning

Authors: Franz Király, Paul Larsen

Abstract: We investigate Fano schemes of conditionally generic intersections, i.e. of hypersurfaces in projective space chosen generically up to additional conditions. Via a correspondence between generic properties of algebraic varieties and events in probability spaces that occur with probability one, we use the obtained results on Fano schemes to solve a problem in machine learning. We investigate Fano schemes of conditionally generic intersections, i.e. of hypersurfaces in projective space chosen generically up to additional conditions. Via a correspondence between generic properties of algebraic varieties and events in probability spaces that occur with probability one, we use the obtained results on Fano schemes to solve a problem in machine learning. △ Less

Submitted 14 January, 2013; originally announced January 2013.

Comments: 9 pages, comments very welcome!

MSC Class: 14J40; 14J70; 14M15; 62M10

Approximate Rank-Detecting Factorization of Low-Rank Tensors

Authors: Franz J. Király, Andreas Ziehe

Abstract: We present an algorithm, AROFAC2, which detects the (CP-)rank of a degree 3 tensor and calculates its factorization into rank-one components. We provide generative conditions for the algorithm to work and demonstrate on both synthetic and real world data that AROFAC2 is a potentially outperforming alternative to the gold standard PARAFAC over which it has the advantages that it can intrinsically d… ▽ More We present an algorithm, AROFAC2, which detects the (CP-)rank of a degree 3 tensor and calculates its factorization into rank-one components. We provide generative conditions for the algorithm to work and demonstrate on both synthetic and real world data that AROFAC2 is a potentially outperforming alternative to the gold standard PARAFAC over which it has the advantages that it can intrinsically detect the true rank, avoids spurious components, and is stable with respect to outliers and non-Gaussian noise. △ Less

Submitted 30 November, 2012; originally announced November 2012.

The Algebraic Combinatorial Approach for Low-Rank Matrix Completion

Authors: Franz J. Király, Louis Theran, Ryota Tomioka

Abstract: We present a novel algebraic combinatorial view on low-rank matrix completion based on studying relations between a few entries with tools from algebraic geometry and matroid theory. The intrinsic locality of the approach allows for the treatment of single entries in a closed theoretical and practical framework. More specifically, apart from introducing an algebraic combinatorial theory of low-ran… ▽ More We present a novel algebraic combinatorial view on low-rank matrix completion based on studying relations between a few entries with tools from algebraic geometry and matroid theory. The intrinsic locality of the approach allows for the treatment of single entries in a closed theoretical and practical framework. More specifically, apart from introducing an algebraic combinatorial theory of low-rank matrix completion, we present probability-one algorithms to decide whether a particular entry of the matrix can be completed. We also describe methods to complete that entry from a few others, and to estimate the error which is incurred by any method completing that entry. Furthermore, we show how known results on matrix completion and their sampling assumptions can be related to our new perspective and interpreted in terms of a completability phase transition. △ Less

Submitted 19 August, 2014; v1 submitted 17 November, 2012; originally announced November 2012.

Comments: 37 pages, with an appendix by Takeaki Uno

A Combinatorial Algebraic Approach for the Identifiability of Low-Rank Matrix Completion

Authors: Franz Kiraly, Ryota Tomioka

Abstract: In this paper, we review the problem of matrix completion and expose its intimate relations with algebraic geometry, combinatorics and graph theory. We present the first necessary and sufficient combinatorial conditions for matrices of arbitrary rank to be identifiable from a set of matrix entries, yielding theoretical constraints and new algorithms for the problem of matrix completion. We conclud… ▽ More In this paper, we review the problem of matrix completion and expose its intimate relations with algebraic geometry, combinatorics and graph theory. We present the first necessary and sufficient combinatorial conditions for matrices of arbitrary rank to be identifiable from a set of matrix entries, yielding theoretical constraints and new algorithms for the problem of matrix completion. We conclude by algorithmically evaluating the tightness of the given conditions and algorithms for practically relevant matrix sizes, showing that the algebraic-combinatoric approach can lead to improvements over state-of-the-art matrix completion methods. △ Less

Submitted 27 June, 2012; originally announced June 2012.

Comments: Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012)

Invariants of normal local rings by p-cyclic group actions

Authors: Franz J. Király, Werner Lütkebohmert

Abstract: Let $B$ be a Noetherian normal local ring, and $G\subset\Aut(B)$ a cyclic group of local automorphisms of prime order. Let $A$ be the ring of $G$-invariants of $B$, assume that $A$ is Noetherian. We study the invariant morphism; in particular, we prove that $B$ is a monogenous $A$-algebra if and only if the augmentation ideal of $B$ is principal. If in particular $B$ is regular, we prove that $A$… ▽ More Let $B$ be a Noetherian normal local ring, and $G\subset\Aut(B)$ a cyclic group of local automorphisms of prime order. Let $A$ be the ring of $G$-invariants of $B$, assume that $A$ is Noetherian. We study the invariant morphism; in particular, we prove that $B$ is a monogenous $A$-algebra if and only if the augmentation ideal of $B$ is principal. If in particular $B$ is regular, we prove that $A$ is regular if the augmentation ideal of $B$ is principal. △ Less

Submitted 4 August, 2011; v1 submitted 12 January, 2010; originally announced January 2010.

MSC Class: 13A50 (Primary); 14L30 (Secondary)

Journal ref: Algebra and Number Theory, Vol.7, No.1, 63-74. 2013