MMTEB: Massive Multilingual Text Embedding Benchmark
Authors:
Kenneth Enevoldsen,
Isaac Chung,
Imene Kerboua,
Márton Kardos,
Ashwin Mathur,
David Stap,
Jay Gala,
Wissam Siblini,
Dominik Krzemiński,
Genta Indra Winata,
Saba Sturua,
Saiteja Utpala,
Mathieu Ciancone,
Marion Schaeffer,
Gabriel Sequeira,
Diganta Misra,
Shreeya Dhakal,
Jonathan Rystrøm,
Roman Solomatin,
Ömer Çağatan,
Akash Kundu,
Martin Bernstorff,
Shitao Xiao,
Akshita Sukhlecha,
Bhavish Pahwa
, et al. (61 additional authors not shown)
Abstract:
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ langua…
▽ More
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more comprehensive evaluation, we introduce the Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale, community-driven expansion of MTEB, covering over 500 quality-controlled evaluation tasks across 250+ languages. MMTEB includes a diverse set of challenging, novel tasks such as instruction following, long-document retrieval, and code retrieval, representing the largest multilingual collection of evaluation tasks for embedding models to date. Using this collection, we develop several highly multilingual benchmarks, which we use to evaluate a representative set of models. We find that while large language models (LLMs) with billions of parameters can achieve state-of-the-art performance on certain language subsets and task categories, the best-performing publicly available model is multilingual-e5-large-instruct with only 560 million parameters. To facilitate accessibility and reduce computational cost, we introduce a novel downsampling method based on inter-task correlation, ensuring a diverse selection while preserving relative model rankings. Furthermore, we optimize tasks such as retrieval by sampling hard negatives, creating smaller but effective splits. These optimizations allow us to introduce benchmarks that drastically reduce computational demands. For instance, our newly introduced zero-shot English benchmark maintains a ranking order similar to the full-scale version but at a fraction of the computational cost.
△ Less
Submitted 8 June, 2025; v1 submitted 19 February, 2025;
originally announced February 2025.
$S^3$ -- Semantic Signal Separation
Authors:
Márton Kardos,
Jan Kostkan,
Arnault-Quentin Vermillet,
Kristoffer Nielbo,
Kenneth Enevoldsen,
Roberta Rocca
Abstract:
Topic models are useful tools for discovering latent semantic structures in large textual corpora. Recent efforts have been oriented at incorporating contextual representations in topic modeling and have been shown to outperform classical topic models. These approaches are typically slow, volatile, and require heavy preprocessing for optimal results. We present Semantic Signal Separation ($S^3$),…
▽ More
Topic models are useful tools for discovering latent semantic structures in large textual corpora. Recent efforts have been oriented at incorporating contextual representations in topic modeling and have been shown to outperform classical topic models. These approaches are typically slow, volatile, and require heavy preprocessing for optimal results. We present Semantic Signal Separation ($S^3$), a theory-driven topic modeling approach in neural embedding spaces. $S^3$ conceptualizes topics as independent axes of semantic space and uncovers these by decomposing contextualized document embeddings using Independent Component Analysis. Our approach provides diverse and highly coherent topics, requires no preprocessing, and is demonstrated to be the fastest contextual topic model, being, on average, 4.5x faster than the runner-up BERTopic. We offer an implementation of $S^3$, and all contextual baselines, in the Turftopic Python package.
△ Less
Submitted 19 May, 2025; v1 submitted 13 June, 2024;
originally announced June 2024.
Event Flow -- How Events Shaped the Flow of the News, 1950-1995
Authors:
Melvin Wevers,
Jan Kostkan,
Kristoffer L. Nielbo
Abstract:
This article relies on information-theoretic measures to examine how events impacted the news for the period 1950-1995. Moreover, we present a method for event characterization in (unstructured) textual sources, offering a taxonomy of events based on the different ways they impacted the flow of news information. The results give us a better understanding of the relationship between events and thei…
▽ More
This article relies on information-theoretic measures to examine how events impacted the news for the period 1950-1995. Moreover, we present a method for event characterization in (unstructured) textual sources, offering a taxonomy of events based on the different ways they impacted the flow of news information. The results give us a better understanding of the relationship between events and their impact on news sources with varying ideological backgrounds.
△ Less
Submitted 17 September, 2021;
originally announced September 2021.
