A Family of Pretrained Transformer Language Models for Russian
Authors:
Dmitry Zmitrovich,
Alexander Abramov,
Andrey Kalmykov,
Maria Tikhonova,
Ekaterina Taktasheva,
Danil Astafurov,
Mark Baushenko,
Artem Snegirev,
Vitalii Kadulin,
Sergey Markov,
Tatiana Shavrina,
Vladislav Mikhailov,
Alena Fenogenova
Abstract:
Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) archite…
▽ More
Transformer language models (LMs) are fundamental to NLP research methodologies and applications in various languages. However, developing such models specifically for the Russian language has received little attention. This paper introduces a collection of 13 Russian Transformer LMs, which spans encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) architectures. We provide a report on the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we aim to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.
△ Less
Submitted 2 August, 2024; v1 submitted 19 September, 2023;
originally announced September 2023.
