-
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Authors:
Jafar Isbarov,
Arofat Akhundjanova,
Mammad Hajili,
Kavsar Huseynova,
Dmitry Gaynullin,
Anar Rzayev,
Osman Tursun,
Aizirek Turdubaeva,
Ilshat Saetov,
Rinat Kharisov,
Saule Belginova,
Ariana Kenbayeva,
Amina Alisheva,
Abdullatif Köksal,
Samir Rustamov,
Duygu Ataman
Abstract:
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, thes…
▽ More
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However, preparing such benchmarks in high quality native language is often costly and therefore limits the representativeness of evaluation datasets. While recent efforts focused on building more inclusive MMLU benchmarks, these are conventionally built using machine translation from high-resource languages, which may introduce errors and fail to account for the linguistic and cultural intricacies of the target languages. In this paper, we address the lack of native language MMLU benchmark especially in the under-represented Turkic language family with distinct morphosyntactic and cultural characteristics. We propose two benchmarks for Turkic language MMLU: TUMLU is a comprehensive, multilingual, and natively developed language understanding benchmark specifically designed for Turkic languages. It consists of middle- and high-school level questions spanning 11 academic subjects in Azerbaijani, Crimean Tatar, Karakalpak, Kazakh, Tatar, Turkish, Uyghur, and Uzbek. We also present TUMLU-mini, a more concise, balanced, and manually verified subset of the dataset. Using this dataset, we systematically evaluate a diverse range of open and proprietary multilingual large language models (LLMs), including Claude, Gemini, GPT, and LLaMA, offering an in-depth analysis of their performance across different languages, subjects, and alphabets. To promote further research and development in multilingual language understanding, we release TUMLU-mini and all corresponding evaluation scripts.
△ Less
Submitted 13 June, 2025; v1 submitted 16 February, 2025;
originally announced February 2025.
-
Open foundation models for Azerbaijani language
Authors:
Jafar Isbarov,
Kavsar Huseynova,
Elvin Mammadov,
Mammad Hajili,
Duygu Ataman
Abstract:
The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4. While there have been several attempts to develop open foundation models for Azerbaijani, these works have not found their way into common use due to a lack of systemic…
▽ More
The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4. While there have been several attempts to develop open foundation models for Azerbaijani, these works have not found their way into common use due to a lack of systemic benchmarking. This paper encompasses several lines of work that promote open-source foundation models for Azerbaijani. We introduce (1) a large text corpus for Azerbaijani, (2) a family of encoder-only language models trained on this dataset, (3) labeled datasets for evaluating these models, and (4) extensive evaluation that covers all major open-source models with Azerbaijani support.
△ Less
Submitted 19 August, 2024; v1 submitted 2 July, 2024;
originally announced July 2024.
-
A Large-Scale Study of Machine Translation in the Turkic Languages
Authors:
Jamshidbek Mirzakhalov,
Anoop Babu,
Duygu Ataman,
Sherzod Kariev,
Francis Tyers,
Otabek Abduraufov,
Mammad Hajili,
Sardana Ivanova,
Abror Khaytbaev,
Antonio Laverghetta Jr.,
Behzodbek Moydinboyev,
Esra Onal,
Shaxnoza Pulatova,
Ahsan Wahab,
Orhan Firat,
Sriram Chellappan
Abstract:
Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted to build competitive systems. However, there is still a large number of languages that are yet to reap the benefits of NMT. In this paper, we provide the first large-scale case study of the practical application of MT in the Turkic language…
▽ More
Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted to build competitive systems. However, there is still a large number of languages that are yet to reap the benefits of NMT. In this paper, we provide the first large-scale case study of the practical application of MT in the Turkic language family in order to realize the gains of NMT for Turkic languages under high-resource to extremely low-resource scenarios. In addition to presenting an extensive analysis that identifies the bottlenecks towards building competitive systems to ameliorate data scarcity, our study has several key contributions, including, i) a large parallel corpus covering 22 Turkic languages consisting of common public datasets in combination with new datasets of approximately 2 million parallel sentences, ii) bilingual baselines for 26 language pairs, iii) novel high-quality test sets in three different translation domains and iv) human evaluation scores. All models, scripts, and data will be released to the public.
△ Less
Submitted 9 September, 2021;
originally announced September 2021.
