-
Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
Authors:
Andros Tjandra,
Yi-Chiao Wu,
Baishan Guo,
John Hoffman,
Brian Ellis,
Apoorv Vyas,
Bowen Shi,
Sanyuan Chen,
Matt Le,
Nick Zacharov,
Carleigh Wood,
Ann Lee,
Wei-Ning Hsu
Abstract:
The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context. Traditional methods often depend on human listeners for evaluation, leading to inconsistencies and high resource demands. This paper addresses the growing need for automated systems capable of predicting audio…
▽ More
The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context. Traditional methods often depend on human listeners for evaluation, leading to inconsistencies and high resource demands. This paper addresses the growing need for automated systems capable of predicting audio aesthetics without human intervention. Such systems are crucial for applications like data filtering, pseudo-labeling large datasets, and evaluating generative audio models, especially as these models become more sophisticated. In this work, we introduce a novel approach to audio aesthetic evaluation by proposing new annotation guidelines that decompose human listening perspectives into four distinct axes. We develop and train no-reference, per-item prediction models that offer a more nuanced assessment of audio quality. Our models are evaluated against human mean opinion scores (MOS) and existing methods, demonstrating comparable or superior performance. This research not only advances the field of audio aesthetics but also provides open-source models and datasets to facilitate future work and benchmarking. We release our code and pre-trained model at: https://github.com/facebookresearch/audiobox-aesthetics
△ Less
Submitted 7 February, 2025;
originally announced February 2025.
-
Movie Gen: A Cast of Media Foundation Models
Authors:
Adam Polyak,
Amit Zohar,
Andrew Brown,
Andros Tjandra,
Animesh Sinha,
Ann Lee,
Apoorv Vyas,
Bowen Shi,
Chih-Yao Ma,
Ching-Yao Chuang,
David Yan,
Dhruv Choudhary,
Dingkang Wang,
Geet Sethi,
Guan Pang,
Haoyu Ma,
Ishan Misra,
Ji Hou,
Jialiang Wang,
Kiran Jagadeesh,
Kunpeng Li,
Luxin Zhang,
Mannat Singh,
Mary Williamson,
Matt Le
, et al. (63 additional authors not shown)
Abstract:
We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization,…
▽ More
We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.
△ Less
Submitted 26 February, 2025; v1 submitted 17 October, 2024;
originally announced October 2024.
-
MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector
Authors:
Marta R. Costa-jussà,
Mariano Coria Meglioli,
Pierre Andrews,
David Dale,
Prangthip Hansanti,
Elahe Kalbassi,
Alex Mourachko,
Christophe Ropers,
Carleigh Wood
Abstract:
Research in toxicity detection in natural language processing for the speech modality (audio-based) is quite limited, particularly for languages other than English. To address these limitations and lay the groundwork for truly multilingual audio-based toxicity detection, we introduce MuTox, the first highly multilingual audio-based dataset with toxicity labels. The dataset comprises 20,000 audio u…
▽ More
Research in toxicity detection in natural language processing for the speech modality (audio-based) is quite limited, particularly for languages other than English. To address these limitations and lay the groundwork for truly multilingual audio-based toxicity detection, we introduce MuTox, the first highly multilingual audio-based dataset with toxicity labels. The dataset comprises 20,000 audio utterances for English and Spanish, and 4,000 for the other 19 languages. To demonstrate the quality of this dataset, we trained the MuTox audio-based toxicity classifier, which enables zero-shot toxicity detection across a wide range of languages. This classifier outperforms existing text-based trainable classifiers by more than 1% AUC, while expanding the language coverage more than tenfold. When compared to a wordlist-based classifier that covers a similar number of languages, MuTox improves precision and recall by approximately 2.5 times. This significant improvement underscores the potential of MuTox in advancing the field of audio-based toxicity detection.
△ Less
Submitted 27 June, 2024; v1 submitted 10 January, 2024;
originally announced January 2024.
-
Audiobox: Unified Audio Generation with Natural Language Prompts
Authors:
Apoorv Vyas,
Bowen Shi,
Matthew Le,
Andros Tjandra,
Yi-Chiao Wu,
Baishan Guo,
Jiemin Zhang,
Xinyue Zhang,
Robert Adkins,
William Ngan,
Jeff Wang,
Ivan Cruz,
Bapi Akula,
Akinniyi Akinyemi,
Brian Ellis,
Rashel Moritz,
Yael Yungster,
Alice Rakotoarison,
Liang Tan,
Chris Summers,
Carleigh Wood,
Joshua Lane,
Mary Williamson,
Wei-Ning Hsu
Abstract:
Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in sever…
▽ More
Audio is an essential part of our life, but creating it often requires expertise and is time-consuming. Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data. However, these models lack controllability in several aspects: speech generation models cannot synthesize novel styles based on text description and are limited on domain coverage such as outdoor environments; sound generation models only provide coarse-grained control based on descriptions like "a person speaking" and would only generate mumbling human voices. This paper presents Audiobox, a unified model based on flow-matching that is capable of generating various audio modalities. We design description-based and example-based prompting to enhance controllability and unify speech and sound generation paradigms. We allow transcript, vocal, and other audio styles to be controlled independently when generating speech. To improve model generalization with limited labels, we adapt a self-supervised infilling objective to pre-train on large quantities of unlabeled audio. Audiobox sets new benchmarks on speech and sound generation (0.745 similarity on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and unlocks new methods for generating audio with novel vocal and acoustic styles. We further integrate Bespoke Solvers, which speeds up generation by over 25 times compared to the default ODE solver for flow-matching, without loss of performance on several tasks. Our demo is available at https://audiobox.metademolab.com/
△ Less
Submitted 25 December, 2023;
originally announced December 2023.
-
Seamless: Multilingual Expressive and Streaming Speech Translation
Authors:
Seamless Communication,
Loïc Barrault,
Yu-An Chung,
Mariano Coria Meglioli,
David Dale,
Ning Dong,
Mark Duppenthaler,
Paul-Ambroise Duquenne,
Brian Ellis,
Hady Elsahar,
Justin Haaheim,
John Hoffman,
Min-Jae Hwang,
Hirofumi Inaguma,
Christopher Klaiber,
Ilia Kulikov,
Pengwei Li,
Daniel Licht,
Jean Maillard,
Ruslan Mavlyutov,
Alice Rakotoarison,
Kaushik Ram Sadagopan,
Abinesh Ramakrishnan,
Tuan Tran,
Guillaume Wenzek
, et al. (40 additional authors not shown)
Abstract:
Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4…
▽ More
Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication
△ Less
Submitted 8 December, 2023;
originally announced December 2023.
-
Algorithms for Least-Squares Noncartesian MR Image Reconstruction
Authors:
Tobias C Wood
Abstract:
Iterative least-squares MR reconstructions typically use the Conjugate Gradient algorithm, despite known numerical issues. This paper demonstrates that the more recent LSMR algorithm has favourable numerical properties, and is to be preferred in situations where Toeplitz embedding cannot be used to accelerate the Conjugate Gradient method.
Iterative least-squares MR reconstructions typically use the Conjugate Gradient algorithm, despite known numerical issues. This paper demonstrates that the more recent LSMR algorithm has favourable numerical properties, and is to be preferred in situations where Toeplitz embedding cannot be used to accelerate the Conjugate Gradient method.
△ Less
Submitted 13 December, 2022;
originally announced December 2022.
-
A generative adversarial approach to facilitate archival-quality histopathologic diagnoses from frozen tissue sections
Authors:
Kianoush Falahkheirkhah,
Tao Guo,
Michael Hwang,
Pheroze Tamboli,
Christopher G Wood,
Jose A Karam,
Kanishka Sircar,
Rohit Bhargava
Abstract:
In clinical diagnostics and research involving histopathology, formalin fixed paraffin embedded (FFPE) tissue is almost universally favored for its superb image quality. However, tissue processing time (more than 24 hours) can slow decision-making. In contrast, fresh frozen (FF) processing (less than 1 hour) can yield rapid information but diagnostic accuracy is suboptimal due to lack of clearing,…
▽ More
In clinical diagnostics and research involving histopathology, formalin fixed paraffin embedded (FFPE) tissue is almost universally favored for its superb image quality. However, tissue processing time (more than 24 hours) can slow decision-making. In contrast, fresh frozen (FF) processing (less than 1 hour) can yield rapid information but diagnostic accuracy is suboptimal due to lack of clearing, morphologic deformation and more frequent artifacts. Here, we bridge this gap using artificial intelligence. We synthesize FFPE-like images ,virtual FFPE, from FF images using a generative adversarial network (GAN) from 98 paired kidney samples derived from 40 patients. Five board-certified pathologists evaluated the results in a blinded test. Image quality of the virtual FFPE data was assessed to be high and showed a close resemblance to real FFPE images. Clinical assessments of disease on the virtual FFPE images showed a higher inter-observer agreement compared to FF images. The nearly instantaneously generated virtual FFPE images can not only reduce time to information but can facilitate more precise diagnosis from routine FF images without extraneous costs and effort.
△ Less
Submitted 24 August, 2021;
originally announced August 2021.
