-
Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora
Authors:
Svanhvít Lilja Ingólfsdóttir,
Pétur Orri Ragnarsson,
Haukur Páll Jónsson,
Haukur Barri Símonarson,
Vilhjálmur Þorsteinsson,
Vésteinn Snæbjarnarson
Abstract:
Grammatical error correction (GEC) is the task of correcting typos, spelling, punctuation and grammatical issues in text. Approaching the problem as a sequence-to-sequence task, we compare the use of a common subword unit vocabulary and byte-level encoding. Initial synthetic training data is created using an error-generating pipeline, and used for finetuning two subword-level models and one byte-l…
▽ More
Grammatical error correction (GEC) is the task of correcting typos, spelling, punctuation and grammatical issues in text. Approaching the problem as a sequence-to-sequence task, we compare the use of a common subword unit vocabulary and byte-level encoding. Initial synthetic training data is created using an error-generating pipeline, and used for finetuning two subword-level models and one byte-level model. Models are then finetuned further on hand-corrected error corpora, including texts written by children, university students, dyslexic and second-language writers, and evaluated over different error types and origins. We show that a byte-level model enables higher correction quality than a subword approach, not only for simple spelling errors, but also for more complex semantic, stylistic and grammatical issues. In particular, initial training on synthetic corpora followed by finetuning on a relatively small parallel corpus of real-world errors helps the byte-level model correct a wide range of commonly occurring errors. Our experiments are run for the Icelandic language but should hold for other similar languages, particularly morphologically rich ones.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models
Authors:
Vésteinn Snæbjarnarson,
Haukur Barri Símonarson,
Pétur Orri Ragnarsson,
Svanhvít Lilja Ingólfsdóttir,
Haukur Páll Jónsson,
Vilhjálmur Þorsteinsson,
Hafsteinn Einarsson
Abstract:
We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts fo…
▽ More
We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts found online by targeting the Icelandic top-level-domain (TLD). Several other public data sources are also collected for a total of 16GB of Icelandic text. To enhance the evaluation of model performance and to raise the bar in baselines for Icelandic, we translate and adapt the WinoGrande dataset for co-reference resolution. Through these efforts we demonstrate that a properly cleaned crawled corpus is sufficient to achieve state-of-the-art results in NLP applications for low to medium resource languages, by comparison with models trained on a curated corpus. We further show that initializing models using existing multilingual models can lead to state-of-the-art results for some downstream tasks.
△ Less
Submitted 18 January, 2022; v1 submitted 14 January, 2022;
originally announced January 2022.
-
Miðeind's WMT 2021 submission
Authors:
Haukur Barri Símonarson,
Vésteinn Snæbjarnarson,
Pétur Orri Ragnarsson,
Haukur Páll Jónsson,
Vilhjálmur Þorsteinsson
Abstract:
We present Miðeind's submission for the English$\to$Icelandic and Icelandic$\to$English subsets of the 2021 WMT news translation task. Transformer-base models are trained for translation on parallel data to generate backtranslations iteratively. A pretrained mBART-25 model is then adapted for translation using parallel data as well as the last backtranslation iteration. This adapted pretrained mod…
▽ More
We present Miðeind's submission for the English$\to$Icelandic and Icelandic$\to$English subsets of the 2021 WMT news translation task. Transformer-base models are trained for translation on parallel data to generate backtranslations iteratively. A pretrained mBART-25 model is then adapted for translation using parallel data as well as the last backtranslation iteration. This adapted pretrained model is then used to re-generate backtranslations, and the training of the adapted model is continued.
△ Less
Submitted 15 September, 2021;
originally announced September 2021.
