-
Inference of a Dyadic Measure and its Simplicial Geometry from Binary Feature Data and Application to Data Quality
Authors:
Linda Ness
Abstract:
We propose a new method for representing data sets with a set of binary feature functions. We compute both the dyadic set structure determined by an order on the binary features together with the canonical product coefficient parameters for the associated dyadic measure and a variant of a nerve simplicial complex determined by the support of the dyadic measure together with its betti numbers. The…
▽ More
We propose a new method for representing data sets with a set of binary feature functions. We compute both the dyadic set structure determined by an order on the binary features together with the canonical product coefficient parameters for the associated dyadic measure and a variant of a nerve simplicial complex determined by the support of the dyadic measure together with its betti numbers. The product coefficient parameters characterize the relative skewness of the dyadic measure at dyadic scales and localities. The more abstract betti number statistics summarize the simplicial geometry of the support of the measure and satisfy a differential privacy property. Both types of statistics can be computed algorithmically from the binary feature representation of the data. This representation provides a new method for pre-processing data into automatically generated features which explicitly characterize the dyadic statistics and geometry of the data. useful for statistical fusion, decision-making, inference, multi-scale hypothesis testing and visualization. We illustrated the methods on a data quality data set. We exploit a representation lemma for dyadic measures on the unit interval (Fefferman, Kenig and Pipher) reformulated for measures on dyadic sets by Bassu, Jones, Ness and Shallcross. We prove that dyadic sets with dyadic measures have a canonical set of binary features and determine canonical nerve simplicial complexes. We compare our methods with other results for measures on sets with tree structures, recent multi-resolution theory, and persistent homology and suggest links to differential privacy, Bayesian reasoning and algebraic statistics.
△ Less
Submitted 29 April, 2017;
originally announced May 2017.
-
Product Formalisms for Measures on Spaces with Binary Tree Structures: Representation, Visualization, and Multiscale Noise
Authors:
Devasis Bassu,
Peter W. Jones,
Linda Ness,
David Shallcross
Abstract:
In this paper we present a theoretical foundation for a representation of a data set as a measure in a very large hierarchically parametrized family of positive measures, whose parameters can be computed explicitly (rather than estimated by optimization), and illustrate its applicability to a wide range of data types. The pre-processing step then consists of representing data sets as simple measur…
▽ More
In this paper we present a theoretical foundation for a representation of a data set as a measure in a very large hierarchically parametrized family of positive measures, whose parameters can be computed explicitly (rather than estimated by optimization), and illustrate its applicability to a wide range of data types. The pre-processing step then consists of representing data sets as simple measures. The theoretical foundation consists of a dyadic product formula representation lemma, a visualization theorem. We also define an additive multiscale noise model which can be used to sample from dyadic measures and a more general multiplicative multiscale noise model which can be used to perturb continuous functions, Borel measures, and dyadic measures. The first two results are based on theorems. The representation uses the very simple concept of a dyadic tree, and hence is widely applicable, easily understood, and easily computed. Since the data sample is represented as a measure, subsequent analysis can exploit statistical and measure theoretic concepts and theories. Because the representation uses the very simple concept of a dyadic tree defined on the universe of a data set and the parameters are simply and explicitly computable and easily interpretable and visualizable, we hope that this approach will be broadly useful to mathematicians, statisticians, and computer scientists who are intrigued by or involved in data science including its mathematical foundations.
△ Less
Submitted 14 June, 2020; v1 submitted 12 January, 2016;
originally announced January 2016.
-
Multi-Scale Local Shape Analysis and Feature Selection in Machine Learning Applications
Authors:
Paul Bendich,
Ellen Gasparovic,
John Harer,
Rauf Izmailov,
Linda Ness
Abstract:
We introduce a method called multi-scale local shape analysis, or MLSA, for extracting features that describe the local structure of points within a dataset. The method uses both geometric and topological features at multiple levels of granularity to capture diverse types of local information for subsequent machine learning algorithms operating on the dataset. Using synthetic and real dataset exam…
▽ More
We introduce a method called multi-scale local shape analysis, or MLSA, for extracting features that describe the local structure of points within a dataset. The method uses both geometric and topological features at multiple levels of granularity to capture diverse types of local information for subsequent machine learning algorithms operating on the dataset. Using synthetic and real dataset examples, we demonstrate significant performance improvement of classification algorithms constructed for these datasets with correspondingly augmented features.
△ Less
Submitted 12 October, 2014;
originally announced October 2014.
