GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
Authors:
GigaChat team,
Mamedov Valentin,
Evgenii Kosarev,
Gregory Leleytner,
Ilya Shchuckin,
Valeriy Berezovskiy,
Daniil Smirnov,
Dmitry Kozlov,
Sergei Averkiev,
Lukyanenko Ivan,
Aleksandr Proshunin,
Ainur Israfilova,
Ivan Baskov,
Artem Chervyakov,
Emil Shakirov,
Mikhail Kolesov,
Daria Khomich,
Darya Latortseva,
Sergei Porkhun,
Yury Fedorov,
Oleg Kutuzov,
Polina Kudriavtseva,
Sofiia Soldatova,
Kolodin Egor,
Stanislav Pyatkin
, et al. (9 additional authors not shown)
Abstract:
Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, includi…
▽ More
Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.
△ Less
Submitted 11 June, 2025;
originally announced June 2025.
Probabilistically Robust Watermarking of Neural Networks
Authors:
Mikhail Pautov,
Nikita Bogdanov,
Stanislav Pyatkin,
Oleg Rogov,
Ivan Oseledets
Abstract:
As deep learning (DL) models are widely and effectively used in Machine Learning as a Service (MLaaS) platforms, there is a rapidly growing interest in DL watermarking techniques that can be used to confirm the ownership of a particular model. Unfortunately, these methods usually produce watermarks susceptible to model stealing attacks. In our research, we introduce a novel trigger set-based water…
▽ More
As deep learning (DL) models are widely and effectively used in Machine Learning as a Service (MLaaS) platforms, there is a rapidly growing interest in DL watermarking techniques that can be used to confirm the ownership of a particular model. Unfortunately, these methods usually produce watermarks susceptible to model stealing attacks. In our research, we introduce a novel trigger set-based watermarking approach that demonstrates resilience against functionality stealing attacks, particularly those involving extraction and distillation. Our approach does not require additional model training and can be applied to any model architecture. The key idea of our method is to compute the trigger set, which is transferable between the source model and the set of proxy models with a high probability. In our experimental study, we show that if the probability of the set being transferable is reasonably high, it can be effectively used for ownership verification of the stolen model. We evaluate our method on multiple benchmarks and show that our approach outperforms current state-of-the-art watermarking techniques in all considered experimental setups.
△ Less
Submitted 18 September, 2024; v1 submitted 16 January, 2024;
originally announced January 2024.
