Salamandra Technical Report
Authors:
Aitor Gonzalez-Agirre,
Marc Pàmies,
Joan Llop,
Irene Baucells,
Severino Da Dalt,
Daniel Tamayo,
José Javier Saiz,
Ferran Espuña,
Jaume Prats,
Javier Aula-Blasco,
Mario Mina,
Iñigo Pikabea,
Adrián Rubio,
Alexander Shvets,
Anna Sallés,
Iñaki Lacunza,
Jorge Palomar,
Júlia Falcão,
Lucía Tormo,
Luis Vasquez-Reina,
Montserrat Marimon,
Oriol Pareras,
Valle Ruiz-Fernández,
Marta Villegas
Abstract:
This work introduces Salamandra, a suite of open-source decoder-only large language models available in three different sizes: 2, 7, and 40 billion parameters. The models were trained from scratch on highly multilingual data that comprises text in 35 European languages and code. Our carefully curated corpus is made exclusively from open-access data compiled from a wide variety of sources. Along wi…
▽ More
This work introduces Salamandra, a suite of open-source decoder-only large language models available in three different sizes: 2, 7, and 40 billion parameters. The models were trained from scratch on highly multilingual data that comprises text in 35 European languages and code. Our carefully curated corpus is made exclusively from open-access data compiled from a wide variety of sources. Along with the base models, supplementary checkpoints that were fine-tuned on public-domain instruction data are also released for chat applications. Additionally, we also share our preliminary experiments on multimodality, which serve as proof-of-concept to showcase potential applications for the Salamandra family. Our extensive evaluations on multilingual benchmarks reveal that Salamandra has strong capabilities, achieving competitive performance when compared to similarly sized open-source models. We provide comprehensive evaluation results both on standard downstream tasks as well as key aspects related to bias and safety.With this technical report, we intend to promote open science by sharing all the details behind our design choices, data curation strategy and evaluation methodology. In addition to that, we deviate from the usual practice by making our training and evaluation scripts publicly accessible. We release all models under a permissive Apache 2.0 license in order to foster future research and facilitate commercial use, thereby contributing to the open-source ecosystem of large language models.
△ Less
Submitted 13 February, 2025; v1 submitted 12 February, 2025;
originally announced February 2025.
