-
Space-Efficient Bimachine Construction Based on the Equalizer Accumulation Principle
Authors:
Stefan Gerdjikov,
Stoyan Mihov,
Klaus U. Schulz
Abstract:
Algorithms for building bimachines from functional transducers found in the literature in a run of the bimachine imitate one successful path of the input transducer. Each single bimachine output exactly corresponds to the output of a single transducer transition. Here we introduce an alternative construction principle where bimachine steps take alternative parallel transducer paths into account, m…
▽ More
Algorithms for building bimachines from functional transducers found in the literature in a run of the bimachine imitate one successful path of the input transducer. Each single bimachine output exactly corresponds to the output of a single transducer transition. Here we introduce an alternative construction principle where bimachine steps take alternative parallel transducer paths into account, maximizing the possible output at each step using a joint view. The size of both the deterministic left and right automaton of the bimachine is restricted by $2^{\vert Q\vert}$ where $\vert Q\vert$ is the number of transducer states. Other bimachine constructions lead to larger subautomata. As a concrete example we present a class of real-time functional transducers with $n+2$ states for which the standard bimachine construction generates a bimachine with at least $Θ(n!)$ states whereas the construction based on the equalizer accumulation principle leads to $2^n + n +3$ states. Our construction can be applied to rational functions from free monoids to "mge monoids", a large class of monoids including free monoids, groups, and others that is closed under Cartesian products.
△ Less
Submitted 27 February, 2018;
originally announced March 2018.
-
Automatic quality evaluation and (semi-) automatic improvement of OCR models for historical printings
Authors:
U. Springmann,
F. Fink,
K. U. Schulz
Abstract:
Good OCR results for historical printings rely on the availability of recognition models trained on diplomatic transcriptions as ground truth, which is both a scarce resource and time-consuming to generate. Instead of having to train a separate model for each historical typeface, we propose a strategy to start from models trained on a combined set of available transcriptions in a variety of fonts.…
▽ More
Good OCR results for historical printings rely on the availability of recognition models trained on diplomatic transcriptions as ground truth, which is both a scarce resource and time-consuming to generate. Instead of having to train a separate model for each historical typeface, we propose a strategy to start from models trained on a combined set of available transcriptions in a variety of fonts. These \emph{mixed models} result in character accuracy rates over 90\% on a test set of printings from the same period of time, but without any representation in the training data, demonstrating the possibility to overcome the typography barrier by generalizing from a few typefaces to a larger set of (similar) fonts in use over a period of time. The output of these mixed models is then used as a baseline to be further improved by both fully automatic methods and semi-automatic methods involving a minimal amount of manual transcriptions. In order to evaluate the recognition quality of each model in a series of models generated during the training process in the absence of any ground truth, we introduce two readily observable quantities that correlate well with true accuracy. These quantities are \emph{mean character confidence C} (as given by the OCR engine OCRopus) and \emph{mean token lexicality L} (a distance measure of OCR tokens from modern wordforms taking historical spelling patterns into account, which can be calculated for any OCR engine). Whereas the fully automatic method is able to improve upon the result of a mixed model by only 1-2 percentage points, already 100-200 hand-corrected lines lead to much better OCR results with character error rates of only a few percent. This procedure minimizes the amount of ground truth production and does not depend on the previous construction of a specific typographic model.
△ Less
Submitted 20 October, 2016; v1 submitted 16 June, 2016;
originally announced June 2016.
-
Corpus analysis without prior linguistic knowledge - unsupervised mining of phrases and subphrase structure
Authors:
Stefan Gerdjikov,
Klaus U. Schulz
Abstract:
When looking at the structure of natural language, "phrases" and "words" are central notions. We consider the problem of identifying such "meaningful subparts" of language of any length and underlying composition principles in a completely corpus-based and language-independent way without using any kind of prior linguistic knowledge. Unsupervised methods for identifying "phrases", mining subphrase…
▽ More
When looking at the structure of natural language, "phrases" and "words" are central notions. We consider the problem of identifying such "meaningful subparts" of language of any length and underlying composition principles in a completely corpus-based and language-independent way without using any kind of prior linguistic knowledge. Unsupervised methods for identifying "phrases", mining subphrase structure and finding words in a fully automated way are described. This can be considered as a step towards automatically computing a "general dictionary and grammar of the corpus". We hope that in the long run variants of our approach turn out to be useful for other kind of sequence data as well, such as, e.g., speech, genom sequences, or music annotation. Even if we are not primarily interested in immediate applications, results obtained for a variety of languages show that our methods are interesting for many practical tasks in text mining, terminology extraction and lexicography, search engine technology, and related fields.
△ Less
Submitted 18 February, 2016;
originally announced February 2016.
-
Good parts first - a new algorithm for approximate search in lexica and string databases
Authors:
Stefan Gerdjikov,
Stoyan Mihov,
Petar Mitankin,
Klaus U. Schulz
Abstract:
We present a new efficient method for approximate search in electronic lexica. Given an input string (the pattern) and a similarity threshold, the algorithm retrieves all entries of the lexicon that are sufficiently similar to the pattern. Search is organized in subsearches that always start with an exact partial match where a substring of the input pattern is aligned with a substring of a lexicon…
▽ More
We present a new efficient method for approximate search in electronic lexica. Given an input string (the pattern) and a similarity threshold, the algorithm retrieves all entries of the lexicon that are sufficiently similar to the pattern. Search is organized in subsearches that always start with an exact partial match where a substring of the input pattern is aligned with a substring of a lexicon word. Afterwards this partial match is extended stepwise to larger substrings. For aligning further parts of the pattern with corresponding parts of lexicon entries, more errors are tolerated at each subsequent step. For supporting this alignment order, which may start at any part of the pattern, the lexicon is represented as a structure that enables immediate access to any substring of a lexicon word and permits the extension of such substrings in both directions. Experimental evaluations of the approximate search procedure are given that show significant efficiency improvements compared to existing techniques. Since the technique can be used for large error bounds it offers interesting possibilities for approximate search in special collections of "long" strings, such as phrases, sentences, or book ti
△ Less
Submitted 3 December, 2015; v1 submitted 4 January, 2013;
originally announced January 2013.
-
Conjunctive Queries over Trees
Authors:
Georg Gottlob,
Christoph Koch,
Klaus U. Schulz
Abstract:
We study the complexity and expressive power of conjunctive queries over unranked labeled trees represented using a variety of structure relations such as ``child'', ``descendant'', and ``following'' as well as unary relations for node labels. We establish a framework for characterizing structures representing trees for which conjunctive queries can be evaluated efficiently. Then we completely c…
▽ More
We study the complexity and expressive power of conjunctive queries over unranked labeled trees represented using a variety of structure relations such as ``child'', ``descendant'', and ``following'' as well as unary relations for node labels. We establish a framework for characterizing structures representing trees for which conjunctive queries can be evaluated efficiently. Then we completely chart the tractability frontier of the problem and establish a dichotomy theorem for our axis relations, i.e., we find all subset-maximal sets of axes for which query evaluation is in polynomial time and show that for all other cases, query evaluation is NP-complete. All polynomial-time results are obtained immediately using the proof techniques from our framework. Finally, we study the expressiveness of conjunctive queries over trees and show that for each conjunctive query, there is an equivalent acyclic positive query (i.e., a set of acyclic conjunctive queries), but that in general this query is not of polynomial size.
△ Less
Submitted 2 February, 2006;
originally announced February 2006.
