-
FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks
Authors:
Min Ma,
Yuma Koizumi,
Shigeki Karita,
Heiga Zen,
Jason Riesa,
Haruko Ishikawa,
Michiel Bacchiani
Abstract:
This paper introduces FLEURS-R, a speech restoration applied version of the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) corpus. FLEURS-R maintains an N-way parallel speech corpus in 102 languages as FLEURS, with improved audio quality and fidelity by applying the speech restoration model Miipher. The aim of FLEURS-R is to advance speech technology in more languages…
▽ More
This paper introduces FLEURS-R, a speech restoration applied version of the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) corpus. FLEURS-R maintains an N-way parallel speech corpus in 102 languages as FLEURS, with improved audio quality and fidelity by applying the speech restoration model Miipher. The aim of FLEURS-R is to advance speech technology in more languages and catalyze research including text-to-speech (TTS) and other speech generation tasks in low-resource languages. Comprehensive evaluations with the restored speech and TTS baseline models trained from the new corpus show that the new corpus obtained significantly improved speech quality while maintaining the semantic contents of the speech. The corpus is publicly released via Hugging Face.
△ Less
Submitted 12 August, 2024;
originally announced August 2024.
-
Multimodal Modeling For Spoken Language Identification
Authors:
Shikhar Bharadwaj,
Min Ma,
Shikhar Vashishth,
Ankur Bapna,
Sriram Ganapathy,
Vera Axelrod,
Siddharth Dalmia,
Wei Han,
Yu Zhang,
Daan van Esch,
Sandy Ritchie,
Partha Talukdar,
Jason Riesa
Abstract:
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language identification task. Prior techniques have been constrained to a single modality; however in the case of video data there is a wealth of other metadata that may be beneficial for this task. In this work, we propose MuSeLI,…
▽ More
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language identification task. Prior techniques have been constrained to a single modality; however in the case of video data there is a wealth of other metadata that may be beneficial for this task. In this work, we propose MuSeLI, a Multimodal Spoken Language Identification method, which delves into the use of various metadata sources to enhance language identification. Our study reveals that metadata such as video title, description and geographic location provide substantial information to identify the spoken language of the multimedia recording. We conduct experiments using two diverse public datasets of YouTube videos, and obtain state-of-the-art results on the language identification task. We additionally conduct an ablation study that describes the distinct contribution of each modality for language recognition.
△ Less
Submitted 19 September, 2023;
originally announced September 2023.
-
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
Authors:
Yu Zhang,
Wei Han,
James Qin,
Yongqiang Wang,
Ankur Bapna,
Zhehuai Chen,
Nanxin Chen,
Bo Li,
Vera Axelrod,
Gary Wang,
Zhong Meng,
Ke Hu,
Andrew Rosenberg,
Rohit Prabhavalkar,
Daniel S. Park,
Parisa Haghani,
Jason Riesa,
Ginger Perng,
Hagen Soltau,
Trevor Strohman,
Bhuvana Ramabhadran,
Tara Sainath,
Pedro Moreno,
Chung-Cheng Chiu,
Johan Schalkwyk
, et al. (2 additional authors not shown)
Abstract:
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quant…
▽ More
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.
△ Less
Submitted 24 September, 2023; v1 submitted 2 March, 2023;
originally announced March 2023.
-
SQuId: Measuring Speech Naturalness in Many Languages
Authors:
Thibault Sellam,
Ankur Bapna,
Joshua Camp,
Diana Mackinnon,
Ankur P. Parikh,
Jason Riesa
Abstract:
Much of text-to-speech research relies on human evaluation, which incurs heavy costs and slows down the development process. The problem is particularly acute in heavily multilingual applications, where recruiting and polling judges can take weeks. We introduce SQuId (Speech Quality Identification), a multilingual naturalness prediction model trained on over a million ratings and tested in 65 loca…
▽ More
Much of text-to-speech research relies on human evaluation, which incurs heavy costs and slows down the development process. The problem is particularly acute in heavily multilingual applications, where recruiting and polling judges can take weeks. We introduce SQuId (Speech Quality Identification), a multilingual naturalness prediction model trained on over a million ratings and tested in 65 locales-the largest effort of this type to date. The main insight is that training one model on many locales consistently outperforms mono-locale baselines. We present our task, the model, and show that it outperforms a competitive baseline based on w2v-BERT and VoiceMOS by 50.0%. We then demonstrate the effectiveness of cross-locale transfer during fine-tuning and highlight its effect on zero-shot locales, i.e., locales for which there is no fine-tuning data. Through a series of analyses, we highlight the role of non-linguistic effects such as sound artifacts in cross-locale transfer. Finally, we present the effect of our design decision, e.g., model size, pre-training diversity, and language rebalancing with several ablation experiments.
△ Less
Submitted 1 June, 2023; v1 submitted 12 October, 2022;
originally announced October 2022.
-
FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
Authors:
Alexis Conneau,
Min Ma,
Simran Khanuja,
Yu Zhang,
Vera Axelrod,
Siddharth Dalmia,
Jason Riesa,
Clara Rivera,
Ankur Bapna
Abstract:
We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark, with approximately 12 hours of speech supervision per language. FLEURS can be used for a variety of speech tasks, including Automatic Speech Recognition (ASR), Speech Languag…
▽ More
We introduce FLEURS, the Few-shot Learning Evaluation of Universal Representations of Speech benchmark. FLEURS is an n-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark, with approximately 12 hours of speech supervision per language. FLEURS can be used for a variety of speech tasks, including Automatic Speech Recognition (ASR), Speech Language Identification (Speech LangID), Translation and Retrieval. In this paper, we provide baselines for the tasks based on multilingual pre-trained models like mSLAM. The goal of FLEURS is to enable speech technology in more languages and catalyze research in low-resource speech understanding.
△ Less
Submitted 24 May, 2022;
originally announced May 2022.
