-
Multimodal Modeling For Spoken Language Identification
Authors:
Shikhar Bharadwaj,
Min Ma,
Shikhar Vashishth,
Ankur Bapna,
Sriram Ganapathy,
Vera Axelrod,
Siddharth Dalmia,
Wei Han,
Yu Zhang,
Daan van Esch,
Sandy Ritchie,
Partha Talukdar,
Jason Riesa
Abstract:
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language identification task. Prior techniques have been constrained to a single modality; however in the case of video data there is a wealth of other metadata that may be beneficial for this task. In this work, we propose MuSeLI,…
▽ More
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance. Conventionally, it is modeled as a speech-based language identification task. Prior techniques have been constrained to a single modality; however in the case of video data there is a wealth of other metadata that may be beneficial for this task. In this work, we propose MuSeLI, a Multimodal Spoken Language Identification method, which delves into the use of various metadata sources to enhance language identification. Our study reveals that metadata such as video title, description and geographic location provide substantial information to identify the spoken language of the multimedia recording. We conduct experiments using two diverse public datasets of YouTube videos, and obtain state-of-the-art results on the language identification task. We additionally conduct an ablation study that describes the distinct contribution of each modality for language recognition.
△ Less
Submitted 19 September, 2023;
originally announced September 2023.
-
Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning
Authors:
Sandy Ritchie,
You-Chi Cheng,
Mingqing Chen,
Rajiv Mathews,
Daan van Esch,
Bo Li,
Khe Chai Sim
Abstract:
Almost none of the 2,000+ languages spoken in Africa have widely available automatic speech recognition systems, and the required data is also only available for a few languages. We have experimented with two techniques which may provide pathways to large vocabulary speech recognition for African languages: multilingual modeling and self-supervised learning. We gathered available open source data…
▽ More
Almost none of the 2,000+ languages spoken in Africa have widely available automatic speech recognition systems, and the required data is also only available for a few languages. We have experimented with two techniques which may provide pathways to large vocabulary speech recognition for African languages: multilingual modeling and self-supervised learning. We gathered available open source data and collected data for 15 languages, and trained experimental models using these techniques. Our results show that pooling the small amounts of data available in multilingual end-to-end models, and pre-training on unsupervised data can help improve speech recognition quality for many African languages.
△ Less
Submitted 4 October, 2022; v1 submitted 5 August, 2022;
originally announced August 2022.
-
Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Data
Authors:
Alëna Aksënova,
Zhehuai Chen,
Chung-Cheng Chiu,
Daan van Esch,
Pavel Golik,
Wei Han,
Levi King,
Bhuvana Ramabhadran,
Andrew Rosenberg,
Suzan Schwartz,
Gary Wang
Abstract:
Building inclusive speech recognition systems is a crucial step towards developing technologies that speakers of all language varieties can use. Therefore, ASR systems must work for everybody independently of the way they speak. To accomplish this goal, there should be available data sets representing language varieties, and also an understanding of model configuration that is the most helpful in…
▽ More
Building inclusive speech recognition systems is a crucial step towards developing technologies that speakers of all language varieties can use. Therefore, ASR systems must work for everybody independently of the way they speak. To accomplish this goal, there should be available data sets representing language varieties, and also an understanding of model configuration that is the most helpful in achieving robust understanding of all types of speech. However, there are not enough data sets for accented speech, and for the ones that are already available, more training approaches need to be explored to improve the quality of accented speech recognition. In this paper, we discuss recent progress towards developing more inclusive ASR systems, namely, the importance of building new data sets representing linguistic diversity, and exploring novel training approaches to improve performance for all users. We address recent directions within benchmarking ASR systems for accented speech, measure the effects of wav2vec 2.0 pre-training on accented speech recognition, and highlight corpora relevant for diverse ASR evaluations.
△ Less
Submitted 16 May, 2022;
originally announced May 2022.
