SMOL: Professionally translated parallel data for 115 under-represented languages
Authors:
Isaac Caswell,
Elizabeth Nielsen,
Jiaming Luo,
Colin Cherry,
Geza Kovacs,
Hadar Shemtov,
Partha Talukdar,
Dinesh Tewari,
Baba Mamadi Diane,
Koulako Moussa Doumbouya,
Djibrila Diane,
Solo Farabado Cissé
Abstract:
We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock translation for low-resource languages (LRLs). SMOL has been translated into 115 under-resourced languages, including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOL-S…
▽ More
We open-source SMOL (Set of Maximal Overall Leverage), a suite of training data to unlock translation for low-resource languages (LRLs). SMOL has been translated into 115 under-resourced languages, including many for which there exist no previous public resources, for a total of 6.1M translated tokens. SMOL comprises two sub-datasets, each carefully chosen for maximum impact given its size: SMOL-Sent, a set of sentences chosen for broad unique token coverage, and SMOL-Doc, a document-level source focusing on a broad topic coverage. They join the already released GATITOS for a trifecta of paragraph, sentence, and token-level content. We demonstrate that using SMOL to prompt or fine-tune Large Language Models yields robust ChrF improvements. In addition to translation, we provide factuality ratings and rationales for all documents in SMOL-Doc, yielding the first factuality datasets for most of these languages.
△ Less
Submitted 17 February, 2025;
originally announced February 2025.
Machine Translation for Nko: Tools, Corpora and Baseline Results
Authors:
Moussa Koulako Bala Doumbouya,
Baba Mamadi Diané,
Solo Farabado Cissé,
Djibrila Diané,
Abdoulaye Sow,
Séré Moussa Doumbouya,
Daouda Bangoura,
Fodé Moriba Bayo,
Ibrahima Sory 2. Condé,
Kalo Mory Diané,
Chris Piech,
Christopher Manning
Abstract:
Currently, there is no usable machine translation system for Nko, a language spoken by tens of millions of people across multiple West African countries, which holds significant cultural and educational value.
To address this issue, we present a set of tools, resources, and baseline results aimed towards the development of usable machine translation systems for Nko and other languages that do no…
▽ More
Currently, there is no usable machine translation system for Nko, a language spoken by tens of millions of people across multiple West African countries, which holds significant cultural and educational value.
To address this issue, we present a set of tools, resources, and baseline results aimed towards the development of usable machine translation systems for Nko and other languages that do not currently have sufficiently large parallel text corpora available.
(1) Fria$\parallel$el: A novel collaborative parallel text curation software that incorporates quality control through copyedit-based workflows.
(2) Expansion of the FLoRes-200 and NLLB-Seed corpora with 2,009 and 6,193 high-quality Nko translations in parallel with 204 and 40 other languages.
(3) nicolingua-0005: A collection of trilingual and bilingual corpora with 130,850 parallel segments and monolingual corpora containing over 3 million Nko words.
(4) Baseline bilingual and multilingual neural machine translation results with the best model scoring 30.83 English-Nko chrF++ on FLoRes-devtest.
△ Less
Submitted 15 November, 2023; v1 submitted 24 October, 2023;
originally announced October 2023.
