Showing 1–37 of 37 results for author: Vitanyi, P M

The cluster structure function

Authors: Andrew R. Cohen, Paul M. B. Vitányi

Abstract: For each partition of a data set into a given number of parts there is a partition such that every part is as much as possible a good model (an "algorithmic sufficient statistic") for the data in that part. Since this can be done for every number between one and the number of data, the result is a function, the cluster structure function. It maps the number of parts of a partition to values relate… ▽ More For each partition of a data set into a given number of parts there is a partition such that every part is as much as possible a good model (an "algorithmic sufficient statistic") for the data in that part. Since this can be done for every number between one and the number of data, the result is a function, the cluster structure function. It maps the number of parts of a partition to values related to the deficiencies of being good models by the parts. Such a function starts with a value at least zero for no partition of the data set and descents to zero for the partition of the data set into singleton parts. The optimal clustering is the one chosen to minimize the cluster structure function. The theory behind the method is expressed in algorithmic information theory (Kolmogorov complexity). In practice the Kolmogorov complexities involved are approximated by a concrete compressor. We give examples using real data sets: the MNIST handwritten digits and the segmentation of real cells as used in stem cell research. △ Less

Submitted 14 October, 2022; v1 submitted 4 January, 2022; originally announced January 2022.

Logical depth for reversible Turing machines with an application to the rate of decrease in logical depth for general Turing machines

Authors: Paul MB Vitanyi

Abstract: The logical depth of a {\em reversible} Turing machine equals the shortest running time of a shortest program for it. This is applied to show that the result in L.F. Antunes, A. Souto, and P.M.B. Vitányi, On the Rate of Decrease in Logical Depth, Theor. Comput. Sci., 702(2017), 60--64 is valid notwithstanding the error noted in Corrigendum P.M.B. Vitányi, Corrigendum to "On the rate of decrease in… ▽ More The logical depth of a {\em reversible} Turing machine equals the shortest running time of a shortest program for it. This is applied to show that the result in L.F. Antunes, A. Souto, and P.M.B. Vitányi, On the Rate of Decrease in Logical Depth, Theor. Comput. Sci., 702(2017), 60--64 is valid notwithstanding the error noted in Corrigendum P.M.B. Vitányi, Corrigendum to "On the rate of decrease in logical depth" by L.F. Antunes, A. Souto, and P.M.B. Vitányi [Theoret. Comput. Sci. 702 (2017) 60--64], {\em Theoret. Comput. Sci.}, https://doi.org/10.1016/j.tcs.2018.07.009 . / △ Less

Submitted 28 August, 2019; originally announced August 2019.

Comments: Latex 4 pages

Journal ref: Theor. Comput. Sci., 778(2019), 78-80

Identification of Probabilities

Authors: Paul M. B. Vitanyi, Nick Chater

Abstract: Within psychology, neuroscience and artificial intelligence, there has been increasing interest in the proposal that the brain builds probabilistic models of sensory and linguistic input: that is, to infer a probabilistic model from a sample. The practical problems of such inference are substantial: the brain has limited data and restricted computational resources. But there is a more fundamental… ▽ More Within psychology, neuroscience and artificial intelligence, there has been increasing interest in the proposal that the brain builds probabilistic models of sensory and linguistic input: that is, to infer a probabilistic model from a sample. The practical problems of such inference are substantial: the brain has limited data and restricted computational resources. But there is a more fundamental question: is the problem of inferring a probabilistic model from a sample possible even in principle? We explore this question and find some surprisingly positive and general results. First, for a broad class of probability distributions characterised by computability restrictions, we specify a learning algorithm that will almost surely identify a probability distribution in the limit given a finite i.i.d. sample of sufficient but unknown length. This is similarly shown to hold for sequences generated by a broad class of Markov chains, subject to computability assumptions. The technical tool is the strong law of large numbers. Second, for a large class of dependent sequences, we specify an algorithm which identifies in the limit a computable measure for which the sequence is typical, in the sense of Martin-Lof (there may be more than one such measure). The technical tool is the theory of Kolmogorov complexity. We analyse the associated predictions in both cases. We also briefly consider special cases, including language learning, and wider theoretical implications for psychology. △ Less

Submitted 4 August, 2017; originally announced August 2017.

Comments: 31 pages LaTeX. arXiv admin note: substantial text overlap with arXiv:1311.7385

Journal ref: Journal of Mathematical Psychology 51, 135-163 (2007)

Web Similarity in Sets of Search Terms using Database Queries

Authors: Andrew R. Cohen, Paul M. B. Vitanyi

Abstract: Normalized web distance (NWD) is a similarity or normalized semantic distance based on the World Wide Web or another large electronic database, for instance Wikipedia, and a search engine that returns reliable aggregate page counts. For sets of search terms the NWD gives a common similarity (common semantics) on a scale from 0 (identical) to 1 (completely different). The NWD approximates the simil… ▽ More Normalized web distance (NWD) is a similarity or normalized semantic distance based on the World Wide Web or another large electronic database, for instance Wikipedia, and a search engine that returns reliable aggregate page counts. For sets of search terms the NWD gives a common similarity (common semantics) on a scale from 0 (identical) to 1 (completely different). The NWD approximates the similarity of members of a set according to all (upper semi)computable properties. We develop the theory and give applications of classifying using Amazon, Wikipedia, and the NCBI website from the National Institutes of Health. The last gives new correlations between health hazards. A restriction of the NWD to a set of two yields the earlier normalized google distance (NGD) but no combination of the NGD's of pairs in a set can extract the information the NWD extracts from the set. The NWD enables a new contextual (different databases) learning approachbased on Kolmogorov complexity theory that incorporates knowledge from these databases. △ Less

Submitted 23 July, 2020; v1 submitted 20 February, 2015; originally announced February 2015.

Comments: LaTeX 18 pages, 3 tables. A precursor is arXiv:1308.3177

Journal ref: SN COMPUT. SCI. 1, 161(2020)

On The Average-Case Complexity of Shellsort

Authors: Paul M. B. Vitanyi

Abstract: We prove a lower bound expressed in the increment sequence on the average-case complexity of the number of inversions of Shellsort. This lower bound is sharp in every case where it could be checked. A special case of this lower bound yields the general Jiang-Li-Vitányi lower bound. We obtain new results e.g. determining the average-case complexity precisely in the Yao-Janson-Knuth 3-pass case. We prove a lower bound expressed in the increment sequence on the average-case complexity of the number of inversions of Shellsort. This lower bound is sharp in every case where it could be checked. A special case of this lower bound yields the general Jiang-Li-Vitányi lower bound. We obtain new results e.g. determining the average-case complexity precisely in the Yao-Janson-Knuth 3-pass case. △ Less

Submitted 8 February, 2017; v1 submitted 26 January, 2015; originally announced January 2015.

Comments: 13 pages LaTeX

Journal ref: Random Structures and Algorithms, 52:2(2018), 354-363

Exact Expression For Information Distance

Authors: P. M. B. Vitanyi

Abstract: Information distance can be defined not only between two strings but also in a finite multiset of strings of cardinality greater than two. We give an elementary proof for expressing the information distance in terms of plain Kolmogorov complexity. It is exact since for each cardinality of the multiset the lower bound for some multiset equals the upper bound for all multisets up to a constant addit… ▽ More Information distance can be defined not only between two strings but also in a finite multiset of strings of cardinality greater than two. We give an elementary proof for expressing the information distance in terms of plain Kolmogorov complexity. It is exact since for each cardinality of the multiset the lower bound for some multiset equals the upper bound for all multisets up to a constant additive term. △ Less

Submitted 11 July, 2017; v1 submitted 27 October, 2014; originally announced October 2014.

Comments: 6 pages LaTeX. added material and corrected it

Journal ref: IEEE Trans. Inform. Theory, 63:8(2017), 4725-4728

A Fast Quartet Tree Heuristic for Hierarchical Clustering

Authors: Rudi L. Cilibrasi, Paul M. B. Vitanyi

Abstract: The Minimum Quartet Tree Cost problem is to construct an optimal weight tree from the $3{n \choose 4}$ weighted quartet topologies on $n$ objects, where optimality means that the summed weight of the embedded quartet topologies is optimal (so it can be the case that the optimal tree embeds all quartets as nonoptimal topologies). We present a Monte Carlo heuristic, based on randomized hill climbing… ▽ More The Minimum Quartet Tree Cost problem is to construct an optimal weight tree from the $3{n \choose 4}$ weighted quartet topologies on $n$ objects, where optimality means that the summed weight of the embedded quartet topologies is optimal (so it can be the case that the optimal tree embeds all quartets as nonoptimal topologies). We present a Monte Carlo heuristic, based on randomized hill climbing, for approximating the optimal weight tree, given the quartet topology weights. The method repeatedly transforms a dendrogram, with all objects involved as leaves, achieving a monotonic approximation to the exact single globally optimal tree. The problem and the solution heuristic has been extensively used for general hierarchical clustering of nontree-like (non-phylogeny) data in various domains and across domains with heterogeneous data. We also present a greatly improved heuristic, reducing the running time by a factor of order a thousand to ten thousand. All this is implemented and available, as part of the CompLearn package. We compare performance and running time of the original and improved versions with those of UPGMA, BioNJ, and NJ, as implemented in the SplitsTree package on genomic data for which the latter are optimized. Keywords: Data and knowledge visualization, Pattern matching--Clustering--Algorithms/Similarity measures, Hierarchical clustering, Global optimization, Quartet tree, Randomized hill-climbing, △ Less

Submitted 12 September, 2014; originally announced September 2014.

Comments: LaTeX, 40 pages, 11 figures; this paper has substantial overlap with arXiv:cs/0606048 in cs.DS

Journal ref: Pattern Recognition, 44 (2011) 662-677

Algorithmic Identification of Probabilities

Authors: Paul M. B. Vitanyi, Nick Chater

Abstract: TThe problem is to identify a probability associated with a set of natural numbers, given an infinite data sequence of elements from the set. If the given sequence is drawn i.i.d. and the probability mass function involved (the target) belongs to a computably enumerable (c.e.) or co-computably enumerable (co-c.e.) set of computable probability mass functions, then there is an algorithm to almost s… ▽ More TThe problem is to identify a probability associated with a set of natural numbers, given an infinite data sequence of elements from the set. If the given sequence is drawn i.i.d. and the probability mass function involved (the target) belongs to a computably enumerable (c.e.) or co-computably enumerable (co-c.e.) set of computable probability mass functions, then there is an algorithm to almost surely identify the target in the limit. The technical tool is the strong law of large numbers. If the set is finite and the elements of the sequence are dependent while the sequence is typical in the sense of Martin-Löf for at least one measure belonging to a c.e. or co-c.e. set of computable measures, then there is an algorithm to identify in the limit a computable measure for which the sequence is typical (there may be more than one such measure). The technical tool is the theory of Kolmogorov complexity. We give the algorithms and consider the associated predictions. △ Less

Submitted 11 July, 2014; v1 submitted 28 November, 2013; originally announced November 2013.

Comments: 19 pages LaTeX.Corrected errors and rewrote the entire paper. arXiv admin note: text overlap with arXiv:1208.5003

On Logical Depth and the Running Time of Shortest Programs

Authors: L. Antunes, A. Souto, P. M. B. Vitanyi

Abstract: The logical depth with significance $b$ of a finite binary string $x$ is the shortest running time of a binary program for $x$ that can be compressed by at most $b$ bits. There is another definition of logical depth. We give two theorems about the quantitative relation between these versions: the first theorem concerns a variation of a known fact with a new proof, the second theorem and its proof… ▽ More The logical depth with significance $b$ of a finite binary string $x$ is the shortest running time of a binary program for $x$ that can be compressed by at most $b$ bits. There is another definition of logical depth. We give two theorems about the quantitative relation between these versions: the first theorem concerns a variation of a known fact with a new proof, the second theorem and its proof are new. We select the above version of logical depth and show the following. There is an infinite sequence of strings of increasing length such that for each $j$ there is a $b$ such that the logical depth of the $j$th string as a function of $j$ is incomputable (it rises faster than any computable function) but with $b$ replaced by $b+1$ the resuling function is computable. Hence the maximal gap between the logical depths resulting from incrementing appropriate $b$'s by 1 rises faster than any computable function. All functions mentioned are upper bounded by the Busy Beaver function. Since for every string its logical depth is nonincreasing in $b$, the minimal computation time of the shortest programs for the sequence of strings as a function of $j$ rises faster than any computable function but not so fast as the Busy Beaver function. △ Less

Submitted 25 October, 2013; originally announced October 2013.

Comments: 12 pages LaTex (this supercedes arXiv:1301.4451)

Normalized Google Distance of Multisets with Applications

Authors: Andrew R. Cohen, P. M. B. Vitanyi

Abstract: Normalized Google distance (NGD) is a relative semantic distance based on the World Wide Web (or any other large electronic database, for instance Wikipedia) and a search engine that returns aggregate page counts. The earlier NGD between pairs of search terms (including phrases) is not sufficient for all applications. We propose an NGD of finite multisets of search terms that is better for many ap… ▽ More Normalized Google distance (NGD) is a relative semantic distance based on the World Wide Web (or any other large electronic database, for instance Wikipedia) and a search engine that returns aggregate page counts. The earlier NGD between pairs of search terms (including phrases) is not sufficient for all applications. We propose an NGD of finite multisets of search terms that is better for many applications. This gives a relative semantics shared by a multiset of search terms. We give applications and compare the results with those obtained using the pairwise NGD. The derivation of NGD method is based on Kolmogorov complexity. △ Less

Submitted 14 August, 2013; originally announced August 2013.

Comments: 25 pages, LaTeX, 3 figures/tables

On the logical depth function

Authors: L. Antunes, A. Souto, A. Teixeira, P. M. B. Vitanyi

Abstract: For a finite binary string $x$ its logical depth $d$ for significance $b$ is the shortest running time of a program for $x$ of length $K(x)+b$. There is another definition of logical depth. We give a new proof that the two versions are close. There is an infinite sequence of strings of consecutive lengths such that for every string there is a $b$ such that incrementing $b$ by 1 makes the associate… ▽ More For a finite binary string $x$ its logical depth $d$ for significance $b$ is the shortest running time of a program for $x$ of length $K(x)+b$. There is another definition of logical depth. We give a new proof that the two versions are close. There is an infinite sequence of strings of consecutive lengths such that for every string there is a $b$ such that incrementing $b$ by 1 makes the associated depths go from incomputable to computable. The maximal gap between depths resulting from incrementing appropriate $b$'s by 1 is incomputable. The size of this gap is upper bounded by the Busy Beaver function. Both the upper and the lower bound hold for the depth with significance 0. As a consequence, the minimal computation time of the associated shortest programs rises faster than any computable function but not so fast as the Busy Beaver function. △ Less

Submitted 5 July, 2013; v1 submitted 18 January, 2013; originally announced January 2013.

Comments: 11 pages LaTeX; previous version was incorrect, this is a new version with almost the same results

Language learning from positive evidence, reconsidered: A simplicity-based approach

Authors: Anne S. Hsu, Nick Chater, Paul M. B. Vitányi

Abstract: Children learn their native language by exposure to their linguistic and communicative environment, but apparently without requiring that their mistakes are corrected. Such learning from positive evidence has been viewed as raising logical problems for language acquisition. In particular, without correction, how is the child to recover from conjecturing an over-general grammar, which will be consi… ▽ More Children learn their native language by exposure to their linguistic and communicative environment, but apparently without requiring that their mistakes are corrected. Such learning from positive evidence has been viewed as raising logical problems for language acquisition. In particular, without correction, how is the child to recover from conjecturing an over-general grammar, which will be consistent with any sentence that the child hears? There have been many proposals concerning how this logical problem can be dissolved. Here, we review recent formal results showing that the learner has sufficient data to learn successfully from positive evidence, if it favours the simplest encoding of the linguistic input. Results include the ability to learn a linguistic prediction, grammaticality judgements, language production, and form-meaning mappings. The simplicity approach can also be scaled-down to analyse the ability to learn a specific linguistic constructions, and is amenable to empirical test as a framework for describing human language acquisition. △ Less

Submitted 18 January, 2013; originally announced January 2013.

Comments: 39 pages, pdf, 1 figure

Journal ref: A.S. Hsu, N. Chater, P.M.B. Vitanyi, Language learning from positive evidence, reconsidered: A simplicity-based approach. Topics in Cognitive Science, 5:1(2013), 35-55

Normalized Compression Distance of Multisets with Applications

Authors: Andrew R. Cohen, Paul M. B. Vitanyi

Abstract: Normalized compression distance (NCD) is a parameter-free, feature-free, alignment-free, similarity measure between a pair of finite objects based on compression. However, it is not sufficient for all applications. We propose an NCD of finite multisets (a.k.a. multiples) of finite objects that is also a metric. Previously, attempts to obtain such an NCD failed. We cover the entire trajectory from… ▽ More Normalized compression distance (NCD) is a parameter-free, feature-free, alignment-free, similarity measure between a pair of finite objects based on compression. However, it is not sufficient for all applications. We propose an NCD of finite multisets (a.k.a. multiples) of finite objects that is also a metric. Previously, attempts to obtain such an NCD failed. We cover the entire trajectory from theoretical underpinning to feasible practice. The new NCD for multisets is applied to retinal progenitor cell classification questions and to related synthetically generated data that were earlier treated with the pairwise NCD. With the new method we achieved significantly better results. Similarly for questions about axonal organelle transport. We also applied the new NCD to handwritten digit recognition and improved classification accuracy significantly over that of pairwise NCD by incorporating both the pairwise and NCD for multisets. In the analysis we use the incomputable Kolmogorov complexity that for practical purposes is approximated from above by the length of the compressed version of the file involved, using a real-world compression program. Index Terms--- Normalized compression distance, multisets or multiples, pattern recognition, data mining, similarity, classification, Kolmogorov complexity, retinal progenitor cells, synthetic data, organelle transport, handwritten character recognition △ Less

Submitted 29 March, 2013; v1 submitted 22 December, 2012; originally announced December 2012.

Comments: LaTeX 28 pages, 3 figures. This version is changed from the preliminary version to the final version. Updates of the theory. How to compute it, special recepies for classification, more applications and better results (see abstract and especially the detailed results in the paper). The title was changed to reflect this. In v4 corrected the proof of Theorem III-7

ACM Class: I.5.3; H.3.3; E.4; J.3

Journal ref: IEEE Trans. Pattern Analysis and Machine Intelligence, 37:8(2015), 1602-1614

Identification of Probabilities of Languages

Authors: Paul M. B. Vitanyi, Nick Chater

Abstract: We consider the problem of inferring the probability distribution associated with a language, given data consisting of an infinite sequence of elements of the languge. We do this under two assumptions on the algorithms concerned: (i) like a real-life algorothm it has round-off errors, and (ii) it has no round-off errors. Assuming (i) we (a) consider a probability mass function of the elements of t… ▽ More We consider the problem of inferring the probability distribution associated with a language, given data consisting of an infinite sequence of elements of the languge. We do this under two assumptions on the algorithms concerned: (i) like a real-life algorothm it has round-off errors, and (ii) it has no round-off errors. Assuming (i) we (a) consider a probability mass function of the elements of the language if the data are drawn independent identically distributed (i.i.d.), provided the probability mass function is computable and has a finite expectation. We give an effective procedure to almost surely identify in the limit the target probability mass function using the Strong Law of Large Numbers. Second (b) we treat the case of possibly incomputable probabilistic mass functions in the above setting. In this case we can only pointswize converge to the target probability mass function almost surely. Third (c) we consider the case where the data are dependent assuming they are typical for at least one computable measure and the language is finite. There is an effective procedure to identify by infinite recurrence a nonempty subset of the computable measures according to which the data is typical. Here we use the theory of Kolmogorov complexity. Assuming (ii) we obtain the weaker result for (a) that the target distribution is identified by infinite recurrence almost surely; (b) stays the same as under assumption (i). We consider the associated predictions. △ Less

Submitted 15 July, 2014; v1 submitted 24 August, 2012; originally announced August 2012.

Comments: 23 pages LaTeX, no pictures 1311.7385 This paper has been withdrawn by the auther due to crucial errors. The same subject is attacked more succesfully with reduced claims in ArXiV 1311.7385

MSC Class: 68

Conditional Kolmogorov Complexity and Universal Probability

Authors: Paul M. B. Vitanyi

Abstract: The Coding Theorem of L.A. Levin connects unconditional prefix Kolmogorov complexity with the discrete universal distribution. There are conditional versions referred to in several publications but as yet there exist no written proofs in English. Here we provide those proofs. They use a different definition than the standard one for the conditional version of the discrete universal distribution. U… ▽ More The Coding Theorem of L.A. Levin connects unconditional prefix Kolmogorov complexity with the discrete universal distribution. There are conditional versions referred to in several publications but as yet there exist no written proofs in English. Here we provide those proofs. They use a different definition than the standard one for the conditional version of the discrete universal distribution. Under the classic definition of conditional probability, there is no conditional version of the Coding Theorem. △ Less

Submitted 22 January, 2013; v1 submitted 5 June, 2012; originally announced June 2012.

Comments: 17 pages (LaTeX); Corrected previous version. arXiv admin note: text overlap with arXiv:cs/0204037

MSC Class: 68Q30; 03D32

Turing Machines and Understanding Computational Complexity

Authors: P. M. B. Vitanyi

Abstract: We describe the Turing Machine, list some of its many influences on the theory of computation and complexity of computations, and illustrate its importance. We describe the Turing Machine, list some of its many influences on the theory of computation and complexity of computations, and illustrate its importance. △ Less

Submitted 5 January, 2012; originally announced January 2012.

Comments: 9 pages, 1 figure, LaTeX. To appear in: Alan Turing - His Work and Impact, Elsevier

Journal ref: In: S. Barry Cooper, Jan van Leeuwen (eds.), "Alan Turing: His Work and Impact", Elsevier, Amsterdam, London, New York, Tokyo, 2013, pp.57-63

Information Distance: New Developments

Authors: P. M. B. Vitanyi

Abstract: In pattern recognition, learning, and data mining one obtains information from information-carrying objects. This involves an objective definition of the information in a single object, the information to go from one object to another object in a pair of objects, the information to go from one object to any other object in a multiple of objects, and the shared information between objects. This is… ▽ More In pattern recognition, learning, and data mining one obtains information from information-carrying objects. This involves an objective definition of the information in a single object, the information to go from one object to another object in a pair of objects, the information to go from one object to any other object in a multiple of objects, and the shared information between objects. This is called "information distance." We survey a selection of new developments in information distance. △ Less

Submitted 5 January, 2012; originally announced January 2012.

Comments: 4 pages, Latex; Series of Publications C, Report C-2011-45, Department of Computer Science, University of Helsinki, pp. 71-74

Journal ref: Proc. 4th Workshop on Information Theoretic Methods in Science and Engineering (WITSME 2011), 2011, pp. 71-74

Compression-based Similarity

Authors: Paul M. B. Vitanyi

Abstract: First we consider pair-wise distances for literal objects consisting of finite binary files. These files are taken to contain all of their meaning, like genomes or books. The distances are based on compression of the objects concerned, normalized, and can be viewed as similarity distances. Second, we consider pair-wise distances between names of objects, like "red" or "christianity." In this case… ▽ More First we consider pair-wise distances for literal objects consisting of finite binary files. These files are taken to contain all of their meaning, like genomes or books. The distances are based on compression of the objects concerned, normalized, and can be viewed as similarity distances. Second, we consider pair-wise distances between names of objects, like "red" or "christianity." In this case the distances are based on searches of the Internet. Such a search can be performed by any search engine that returns aggregate page counts. We can extract a code length from the numbers returned, use the same formula as before, and derive a similarity or relative semantics between names for objects. The theory is based on Kolmogorov complexity. We test both similarities extensively experimentally. △ Less

Submitted 20 October, 2011; originally announced October 2011.

Comments: Latex, 8 pages, 2 fgures, in Proc. IEEE 1st Int. Conf. Data Compression, Communication and Processing, Palurno, Italy, June 21-24, 2011, 111--118

On Empirical Entropy

Authors: Paul M. B. Vitányi

Abstract: We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the description of the random variable involved plus the entropy it induces. We assume only that the distribution involved is computable. To test the new notion we compare th… ▽ More We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the description of the random variable involved plus the entropy it induces. We assume only that the distribution involved is computable. To test the new notion we compare the Normalized Information Distance (the similarity metric) with a related measure based on Mutual Information in Shannon's framework. This way the similarities and differences of the last two concepts are exposed. △ Less

Submitted 30 March, 2011; originally announced March 2011.

Comments: 14 pages, LaTeX

MSC Class: 68; 94 ACM Class: H.1; F.1; J.1

Information Distance

Authors: Charles H. Bennett, Peter Gacs, Ming Li, Paul M. B. Vitanyi, Wojciech H. Zurek

Abstract: While Kolmogorov complexity is the accepted absolute measure of information content in an individual finite object, a similarly absolute notion is needed for the information distance between two individual objects, for example, two pictures. We give several natural definitions of a universal information metric, based on length of shortest programs for either ordinary computations or reversible (di… ▽ More While Kolmogorov complexity is the accepted absolute measure of information content in an individual finite object, a similarly absolute notion is needed for the information distance between two individual objects, for example, two pictures. We give several natural definitions of a universal information metric, based on length of shortest programs for either ordinary computations or reversible (dissipationless) computations. It turns out that these definitions are equivalent up to an additive logarithmic term. We show that the information distance is a universal cognitive similarity distance. We investigate the maximal correlation of the shortest programs involved, the maximal uncorrelation of programs (a generalization of the Slepian-Wolf theorem of classical information theory), and the density properties of the discrete metric spaces induced by the information distances. A related distance measures the amount of nonreversibility of a computation. Using the physical theory of reversible computation, we give an appropriate (universal, anti-symmetric, and transitive) measure of the thermodynamic work required to transform one object in another object by the most efficient process. Information distance between individual objects is needed in pattern recognition where one wants to express effective notions of "pattern similarity" or "cognitive similarity" between individual objects and in thermodynamics of computation where one wants to analyse the energy dissipation of a computation from a particular input to a particular output. △ Less

Submitted 17 June, 2010; originally announced June 2010.

Comments: 39 pages, LaTeX, 2 Figures/Tables

MSC Class: 68Q30; 94A15; 94A17

Journal ref: C.H. Bennett, P. Gács, M. Li, P.M.B. Vitányi, and W. Zurek, Information Distance, IEEE Trans. Information Theory, 44:4(1998) 1407--1423

Normalized Information Distance is Not Semicomputable

Authors: Sebastiaan A. Terwijn, Leen Torenvliet, Paul M. B. Vitanyi

Abstract: Normalized information distance (NID) uses the theoretical notion of Kolmogorov complexity, which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. This practical application is called 'normalized compression distance' and it is trivially computable. It is a parameter-free similarity measure based on compres… ▽ More Normalized information distance (NID) uses the theoretical notion of Kolmogorov complexity, which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. This practical application is called 'normalized compression distance' and it is trivially computable. It is a parameter-free similarity measure based on compression, and is used in pattern recognition, data mining, phylogeny, clustering, and classification. The complexity properties of its theoretical precursor, the NID, have been open. We show that the NID is neither upper semicomputable nor lower semicomputable. △ Less

Submitted 16 June, 2010; originally announced June 2010.

Comments: 9 pages, LaTeX, No figures, To appear in J. Comput. Syst. Sci

MSC Class: 03Dxx; 62B10; 68T10; 91C20

The probabilistic analysis of language acquisition: Theoretical, computational, and experimental analysis

Authors: Anne S. Hsu, Nick Chater, Paul M. B. Vitanyi

Abstract: There is much debate over the degree to which language learning is governed by innate language-specific biases, or acquired through cognition-general principles. Here we examine the probabilistic language acquisition hypothesis on three levels: We outline a novel theoretical result showing that it is possible to learn the exact generative model underlying a wide class of languages, purely from obs… ▽ More There is much debate over the degree to which language learning is governed by innate language-specific biases, or acquired through cognition-general principles. Here we examine the probabilistic language acquisition hypothesis on three levels: We outline a novel theoretical result showing that it is possible to learn the exact generative model underlying a wide class of languages, purely from observing samples of the language. We then describe a recently proposed practical framework, which quantifies natural language learnability, allowing specific learnability predictions to be made for the first time. In previous work, this framework was used to make learnability predictions for a wide variety of linguistic constructions, for which learnability has been much debated. Here, we present a new experiment which tests these learnability predictions. We find that our experimental results support the possibility that these linguistic constructions are acquired probabilistically from cognition-general principles. △ Less

Submitted 16 June, 2010; originally announced June 2010.

Comments: 26 pages, pdf, 4 figures, Submitted to "Cognition"

MSC Class: 91E10; 97C30; 68T50

Nonapproximablity of the Normalized Information Distance

Authors: Sebastiaan A. Terwijn, Leen Torenvliet, Paul M. B. Vitanyi

Abstract: Normalized information distance (NID) uses the theoretical notion of Kolmogorov complexity, which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. This practical application is called `normalized compression distance' and it is trivially computable. It is a parameter-free similarity measure based on compr… ▽ More Normalized information distance (NID) uses the theoretical notion of Kolmogorov complexity, which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. This practical application is called `normalized compression distance' and it is trivially computable. It is a parameter-free similarity measure based on compression, and is used in pattern recognition, data mining, phylogeny, clustering, and classification. The complexity properties of its theoretical precursor, the NID, have been open. We show that the NID is neither upper semicomputable nor lower semicomputable up to any reasonable precision. △ Less

Submitted 23 October, 2009; v1 submitted 22 October, 2009; originally announced October 2009.

Comments: LaTeX 8 pages, Submitted. 2nd version corrected some typos

Distributed elections in an Archimedean ring of processors

Authors: Paul M. B. Vitanyi

Abstract: Unlimited asynchronism is intolerable in real physically distributed computer systems. Such systems, synchronous or not, use clocks and timeouts. Therefore the magnitudes of elapsed absolute time in the system need to satisfy the axiom of Archimedes. Under this restriction of asynchronicity logically time-independent solutions can be derived which are nonetheless better (in number of message pas… ▽ More Unlimited asynchronism is intolerable in real physically distributed computer systems. Such systems, synchronous or not, use clocks and timeouts. Therefore the magnitudes of elapsed absolute time in the system need to satisfy the axiom of Archimedes. Under this restriction of asynchronicity logically time-independent solutions can be derived which are nonetheless better (in number of message passes) than is possible otherwise. The use of clocks by the individual processors, in elections in a ring of asynchronous processors without central control, allows a deterministic solution which requires but a linear number of message passes. To obtain the result it has to be assumed that the clocks measure finitely proportional absolute time-spans for their time units, that is, the magnitudes of elapsed time in the ring network satisfy the axiom of Archimedes. As a result, some basic subtilities associated with distributed computations are highlighted. For instance, the known nonlinear lower bound on the required number of message passes is cracked. For the synchronous case, in which the necessary assumptions hold a fortiori, the method is -asymptotically- the most efficient one yet, and of optimal order of magnitude. The deterministic algorithm is of -asymptotically- optimal bit complexity, and, in the synchronous case, also yields an optimal method to determine the ring size. All of these results improve the known ones. △ Less

Submitted 27 May, 2009; originally announced June 2009.

Journal ref: 16th ACM Symposium on Theory of Computing, Washington D.C., 1984, 542 - 547

Analysis of Sorting Algorithms by Kolmogorov Complexity (A Survey)

Authors: Paul M. B. Vitanyi

Abstract: Recently, many results on the computational complexity of sorting algorithms were obtained using Kolmogorov complexity (the incompressibility method). Especially, the usually hard average-case analysis is ammenable to this method. Here we survey such results about Bubblesort, Heapsort, Shellsort, Dobosiewicz-sort, Shakersort, and sorting with stacks and queues in sequential or parallel mode. Esp… ▽ More Recently, many results on the computational complexity of sorting algorithms were obtained using Kolmogorov complexity (the incompressibility method). Especially, the usually hard average-case analysis is ammenable to this method. Here we survey such results about Bubblesort, Heapsort, Shellsort, Dobosiewicz-sort, Shakersort, and sorting with stacks and queues in sequential or parallel mode. Especially in the case of Shellsort the uses of Kolmogorov complexity surprisingly easily resolved problems that had stayed open for a long time despite strenuous attacks. △ Less

Submitted 27 May, 2009; originally announced May 2009.

Comments: 18 Pages, 2 figures, LaTeX

Journal ref: Pp.209--232 in: In: Entropy, Search, Complexity, Bolyai Society Mathematical Studies, 16, I. Csiszar, G.O.H. Katona, G. Tardos, Eds., Springer-Verlag, 2007

Normalized Web Distance and Word Similarity

Authors: Rudi L. Cilibrasi, Paul M. B. Vitanyi

Abstract: There is a great deal of work in cognitive psychology, linguistics, and computer science, about using word (or phrase) frequencies in context in text corpora to develop measures for word similarity or word association, going back to at least the 1960s. The goal of this chapter is to introduce the normalizedis a general way to tap the amorphous low-grade knowledge available for free on the Intern… ▽ More There is a great deal of work in cognitive psychology, linguistics, and computer science, about using word (or phrase) frequencies in context in text corpora to develop measures for word similarity or word association, going back to at least the 1960s. The goal of this chapter is to introduce the normalizedis a general way to tap the amorphous low-grade knowledge available for free on the Internet, typed in by local users aiming at personal gratification of diverse objectives, and yet globally achieving what is effectively the largest semantic electronic database in the world. Moreover, this database is available for all by using any search engine that can return aggregate page-count estimates for a large range of search-queries. In the paper introducing the NWD it was called `normalized Google distance (NGD),' but since Google doesn't allow computer searches anymore, we opt for the more neutral and descriptive NWD. web distance (NWD) method to determine similarity between words and phrases. It △ Less

Submitted 25 May, 2009; originally announced May 2009.

Comments: Latex, 20 pages, 7 figures, to appear in: Handbook of Natural Language Processing, Second Edition, Nitin Indurkhya and Fred J. Damerau Eds., CRC Press, Taylor and Francis Group, Boca Raton, FL, 2010, ISBN 978-1420085921

Information Distance in Multiples

Authors: Paul M. B. Vitanyi

Abstract: Information distance is a parameter-free similarity measure based on compression, used in pattern recognition, data mining, phylogeny, clustering, and classification. The notion of information distance is extended from pairs to multiples (finite lists). We study maximal overlap, metricity, universality, minimal overlap, additivity, and normalized information distance in multiples. We use the the… ▽ More Information distance is a parameter-free similarity measure based on compression, used in pattern recognition, data mining, phylogeny, clustering, and classification. The notion of information distance is extended from pairs to multiples (finite lists). We study maximal overlap, metricity, universality, minimal overlap, additivity, and normalized information distance in multiples. We use the theoretical notion of Kolmogorov complexity which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. {\em Index Terms}-- Information distance, multiples, pattern recognition, data mining, similarity, Kolmogorov complexity △ Less

Submitted 20 May, 2009; originally announced May 2009.

Comments: LateX 14 pages, Submitted to a technical journal

ACM Class: J.3; E.4

On Time-Bounded Incompressibility of Compressible Strings and Sequences

Authors: E. G. Daylight, W. M. Koolen, P. M. B. Vitanyi

Abstract: For every total recursive time bound $t$, a constant fraction of all compressible (low Kolmogorov complexity) strings is $t$-bounded incompressible (high time-bounded Kolmogorov complexity); there are uncountably many infinite sequences of which every initial segment of length $n$ is compressible to $\log n$ yet $t$-bounded incompressible below ${1/4}n - \log n$; and there are countable infinite… ▽ More For every total recursive time bound $t$, a constant fraction of all compressible (low Kolmogorov complexity) strings is $t$-bounded incompressible (high time-bounded Kolmogorov complexity); there are uncountably many infinite sequences of which every initial segment of length $n$ is compressible to $\log n$ yet $t$-bounded incompressible below ${1/4}n - \log n$; and there are countable infinitely many recursive infinite sequence of which every initial segment is similarly $t$-bounded incompressible. These results are related to, but different from, Barzdins's lemma. △ Less

Submitted 11 August, 2009; v1 submitted 17 September, 2008; originally announced September 2008.

Comments: 9 pages, LaTeX, no figures, submitted to Information Processing Letters. Changed and added a Barzdins-like lemma for infinite sequences with different quantification oreder, a fixed constant, and uncountably many sequences

Algorithmic information theory

Authors: Peter D. Grunwald, Paul M. B. Vitanyi

Abstract: We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to forma… ▽ More We introduce algorithmic information theory, also known as the theory of Kolmogorov complexity. We explain the main concepts of this quantitative approach to defining `information'. We discuss the extent to which Kolmogorov's and Shannon's information theory have a common purpose, and where they are fundamentally different. We indicate how recent developments within the theory allow one to formally distinguish between `structural' (meaningful) and `random' information as measured by the Kolmogorov structure function, which leads to a mathematical formalization of Occam's razor in inductive inference. We end by discussing some of the philosophical implications of the theory. △ Less

Submitted 17 September, 2008; v1 submitted 16 September, 2008; originally announced September 2008.

Comments: 37 pages, 2 figures, pdf, in: Philosophy of Information, P. Adriaans and J. van Benthem, Eds., A volume in Handbook of the philosophy of science, D. Gabbay, P. Thagard, and J. Woods, Eds., Elsevier, 2008. In version 1 of September 16 the refs are missing. Corrected in version 2 of September 17

Normalized Information Distance

Authors: Paul M. B. Vitanyi, Frank J. Balbach, Rudi L. Cilibrasi, Ming Li

Abstract: The normalized information distance is a universal distance measure for objects of all kinds. It is based on Kolmogorov complexity and thus uncomputable, but there are ways to utilize it. First, compression algorithms can be used to approximate the Kolmogorov complexity if the objects have a string representation. Second, for names and abstract concepts, page count statistics from the World Wide… ▽ More The normalized information distance is a universal distance measure for objects of all kinds. It is based on Kolmogorov complexity and thus uncomputable, but there are ways to utilize it. First, compression algorithms can be used to approximate the Kolmogorov complexity if the objects have a string representation. Second, for names and abstract concepts, page count statistics from the World Wide Web can be used. These practical realizations of the normalized information distance can then be applied to machine learning tasks, expecially clustering, to perform feature-free and parameter-free data mining. This chapter discusses the theoretical foundations of the normalized information distance and both practical realizations. It presents numerous examples of successful real-world applications based on these distance measures, ranging from bioinformatics to music clustering to machine translation. △ Less

Submitted 15 September, 2008; originally announced September 2008.

Comments: 33 pages, 12 figures, pdf, in: Normalized information distance, in: Information Theory and Statistical Learning, Eds. M. Dehmer, F. Emmert-Streib, Springer-Verlag, New-York, To appear

Tales of Huffman

Authors: Paul M. B. Vitanyi, Zvi Lotker

Abstract: We study the new problem of Huffman-like codes subject to individual restrictions on the code-word lengths of a subset of the source words. These are prefix codes with minimal expected code-word length for a random source where additionally the code-word lengths of a subset of the source words is prescribed, possibly differently for every such source word. Based on a structural analysis of prope… ▽ More We study the new problem of Huffman-like codes subject to individual restrictions on the code-word lengths of a subset of the source words. These are prefix codes with minimal expected code-word length for a random source where additionally the code-word lengths of a subset of the source words is prescribed, possibly differently for every such source word. Based on a structural analysis of properties of optimal solutions, we construct an efficient dynamic programming algorithm for this problem, and for an integer programming problem that may be of independent interest. △ Less

Submitted 25 December, 2006; originally announced December 2006.

Comments: LaTex 8 pages

Registers

Authors: Paul M. B. Vitanyi

Abstract: Entry in: Encyclopedia of Algorithms, Ming-Yang Kao, Ed., Springer, To appear. Synonyms: Wait-free registers, wait-free shared variables, asynchronous communication hardware. Problem Definition: Consider a system of asynchronous processes that communicate among themselves by only executing read and write operations on a set of shared variables (also known as shared registers). The system has n… ▽ More Entry in: Encyclopedia of Algorithms, Ming-Yang Kao, Ed., Springer, To appear. Synonyms: Wait-free registers, wait-free shared variables, asynchronous communication hardware. Problem Definition: Consider a system of asynchronous processes that communicate among themselves by only executing read and write operations on a set of shared variables (also known as shared registers). The system has no global clock or other synchronization primitives. △ Less

Submitted 5 December, 2006; originally announced December 2006.

Comments: 5 pages, LaTeX, Entry in: Encyclopedia of Algorithms, Ming-Yang Kao, Ed., Springer, To appear

A New Quartet Tree Heuristic for Hierarchical Clustering

Authors: Rudi Cilibrasi, Paul M. B. Vitanyi

Abstract: We consider the problem of constructing an an optimal-weight tree from the 3*(n choose 4) weighted quartet topologies on n objects, where optimality means that the summed weight of the embedded quartet topologiesis optimal (so it can be the case that the optimal tree embeds all quartets as non-optimal topologies). We present a heuristic for reconstructing the optimal-weight tree, and a canonical… ▽ More We consider the problem of constructing an an optimal-weight tree from the 3*(n choose 4) weighted quartet topologies on n objects, where optimality means that the summed weight of the embedded quartet topologiesis optimal (so it can be the case that the optimal tree embeds all quartets as non-optimal topologies). We present a heuristic for reconstructing the optimal-weight tree, and a canonical manner to derive the quartet-topology weights from a given distance matrix. The method repeatedly transforms a bifurcating tree, with all objects involved as leaves, achieving a monotonic approximation to the exact single globally optimal tree. This contrasts to other heuristic search methods from biological phylogeny, like DNAML or quartet puzzling, which, repeatedly, incrementally construct a solution from a random order of objects, and subsequently add agreement values. △ Less

Submitted 11 June, 2006; originally announced June 2006.

Comments: 22 pages, 14 figures

ACM Class: F.2.2; G.1.6

The Google Similarity Distance

Authors: Rudi Cilibrasi, Paul M. B. Vitanyi

Abstract: Words and phrases acquire meaning from the way they are used in society, from their relative semantics to other words and phrases. For computers the equivalent of `society' is `database,' and the equivalent of `use' is `way to search the database.' We present a new theory of similarity between words and phrases based on information distance and Kolmogorov complexity. To fix thoughts we use the w… ▽ More Words and phrases acquire meaning from the way they are used in society, from their relative semantics to other words and phrases. For computers the equivalent of `society' is `database,' and the equivalent of `use' is `way to search the database.' We present a new theory of similarity between words and phrases based on information distance and Kolmogorov complexity. To fix thoughts we use the world-wide-web as database, and Google as search engine. The method is also applicable to other search engines and databases. This theory is then applied to construct a method to automatically extract similarity, the Google similarity distance, of words and phrases from the world-wide-web using Google page counts. The world-wide-web is the largest database on earth, and the context information entered by millions of independent users averages out to provide automatic semantics of useful quality. We give applications in hierarchical clustering, classification, and language translation. We give examples to distinguish between colors and numbers, cluster names of paintings by 17th century Dutch masters and names of books by English novelists, the ability to understand emergencies, and primes, and we demonstrate the ability to do a simple automatic English-Spanish translation. Finally, we use the WordNet database as an objective baseline against which to judge the performance of our method. We conduct a massive randomized trial in binary classification using support vector machines to learn categories based on our Google distance, resulting in an a mean agreement of 87% with the expert crafted WordNet categories. △ Less

Submitted 30 May, 2007; v1 submitted 21 December, 2004; originally announced December 2004.

Comments: 15 pages, 10 figures; changed some text/figures/notation/part of theorem. Incorporated referees comments. This is the final published version up to some minor changes in the galley proofs

ACM Class: I.2.4; I.2.7

Journal ref: R.L. Cilibrasi, P.M.B. Vitanyi, The Google Similarity Distance, IEEE Trans. Knowledge and Data Engineering, 19:3(2007), 370-383

Rate Distortion and Denoising of Individual Data Using Kolmogorov complexity

Authors: Nikolai K. Vereshchagin, Paul M. B. Vitanyi

Abstract: We examine the structure of families of distortion balls from the perspective of Kolmogorov complexity. Special attention is paid to the canonical rate-distortion function of a source word which returns the minimal Kolmogorov complexity of all distortion balls containing that word subject to a bound on their cardinality. This canonical rate-distortion function is related to the more standard alg… ▽ More We examine the structure of families of distortion balls from the perspective of Kolmogorov complexity. Special attention is paid to the canonical rate-distortion function of a source word which returns the minimal Kolmogorov complexity of all distortion balls containing that word subject to a bound on their cardinality. This canonical rate-distortion function is related to the more standard algorithmic rate-distortion function for the given distortion measure. Examples are given of list distortion, Hamming distortion, and Euclidean distortion. The algorithmic rate-distortion function can behave differently from Shannon's rate-distortion function. To this end, we show that the canonical rate-distortion function can and does assume a wide class of shapes (unlike Shannon's); we relate low algorithmic mutual information to low Kolmogorov complexity (and consequently suggest that certain aspects of the mutual information formulation of Shannon's rate-distortion function behave differently than would an analogous formulation using algorithmic mutual information); we explore the notion that low Kolmogorov complexity distortion balls containing a given word capture the interesting properties of that word (which is hard to formalize in Shannon's theory) and this suggests an approach to denoising; and, finally, we show that the different behavior of the rate-distortion curves of individual source words to some extent disappears after averaging over the source words. △ Less

Submitted 26 November, 2009; v1 submitted 6 November, 2004; originally announced November 2004.

Comments: LaTex, 31 pages, 2 figures. The new version is again completely rewritten, newly titled, and adds new results

ACM Class: E.4; H.1.1

Randomness

Authors: Paul M. B. Vitanyi

Abstract: Here we present in a single essay a combination and completion of the several aspects of the problem of randomness of individual objects which of necessity occur scattered in our texbook "An Introduction to Kolmogorov Complexity and Its Applications" (M. Li and P. Vitanyi), 2nd Ed., Springer-Verlag, 1997. Here we present in a single essay a combination and completion of the several aspects of the problem of randomness of individual objects which of necessity occur scattered in our texbook "An Introduction to Kolmogorov Complexity and Its Applications" (M. Li and P. Vitanyi), 2nd Ed., Springer-Verlag, 1997. △ Less

Submitted 10 October, 2001; v1 submitted 8 October, 2001; originally announced October 2001.

Comments: LaTeX source, 48 pages, Section contributed to `Matematica, Logica, Informatica' Volume 12 of the "Storia del XX Secolo", published by the "Instituto della Enciclopedia Italiana" (smal addition in new version)

MSC Class: 60-02; 60A05; 62-02; 62A01

Quantum Kolmogorov Complexity Based on Classical Descriptions

Authors: Paul M. B. Vitanyi

Abstract: We develop a theory of the algorithmic information in bits contained in an individual pure quantum state. This extends classical Kolmogorov complexity to the quantum domain retaining classical descriptions. Quantum Kolmogorov complexity coincides with the classical Kolmogorov complexity on the classical domain. Quantum Kolmogorov complexity is upper bounded and can be effectively approximated fr… ▽ More We develop a theory of the algorithmic information in bits contained in an individual pure quantum state. This extends classical Kolmogorov complexity to the quantum domain retaining classical descriptions. Quantum Kolmogorov complexity coincides with the classical Kolmogorov complexity on the classical domain. Quantum Kolmogorov complexity is upper bounded and can be effectively approximated from above under certain conditions. With high probability a quantum object is incompressible. Upper- and lower bounds of the quantum complexity of multiple copies of individual pure quantum states are derived and may shed some light on the no-cloning properties of quantum states. In the quantum situation complexity is not sub-additive. We discuss some relations with ``no-cloning'' and ``approximate cloning'' properties. △ Less

Submitted 9 October, 2001; v1 submitted 21 February, 2001; originally announced February 2001.

Comments: 17 pages, LaTeX, final and extended version of quant-ph/9907035, with corrections to the published journal version (the two displayed equations in the right-hand column on page 2466 had the left-hand sides of the displayed formulas erroneously interchanged)

Journal ref: IEEE Transactions on Information Theory, Vol. 47, No. 6, September 2001, 2464-2479