-
AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
Authors:
Joseph Roth,
Sourish Chaudhuri,
Ondrej Klejch,
Radhika Marvin,
Andrew Gallagher,
Liat Kaver,
Sharadh Ramaswamy,
Arkadiusz Stopczynski,
Cordelia Schmid,
Zhonghua Xi,
Caroline Pantofaru
Abstract:
Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made com…
▽ More
Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made comparisons and improvements difficult. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) that will be released publicly to facilitate algorithm development and enable comparisons. The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames or about 38.5 hours of face tracks, and the corresponding audio. We also present a new audio-visual approach for active speaker detection, and analyze its performance, demonstrating both its strength and the contributions of the dataset.
△ Less
Submitted 24 May, 2019; v1 submitted 4 January, 2019;
originally announced January 2019.
-
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation
Authors:
Brian Thompson,
Huda Khayrallah,
Antonios Anastasopoulos,
Arya D. McCarthy,
Kevin Duh,
Rebecca Marvin,
Paul McNamee,
Jeremy Gwinnup,
Tim Anderson,
Philipp Koehn
Abstract:
To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component's contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surpri…
▽ More
To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component's contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surprisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitivity analysis metric, suggesting that the out-of-domain model can provide a good generic initialization for the new domain.
△ Less
Submitted 15 January, 2019; v1 submitted 13 September, 2018;
originally announced September 2018.
-
Targeted Syntactic Evaluation of Language Models
Authors:
Rebecca Marvin,
Tal Linzen
Abstract:
We present a dataset for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity…
▽ More
We present a dataset for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one. In an experiment using this data set, an LSTM language model performed poorly on many of the constructions. Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.
△ Less
Submitted 27 August, 2018;
originally announced August 2018.
-
AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies
Authors:
Sourish Chaudhuri,
Joseph Roth,
Daniel P. W. Ellis,
Andrew Gallagher,
Liat Kaver,
Radhika Marvin,
Caroline Pantofaru,
Nathan Reale,
Loretta Guarino Reid,
Kevin Wilson,
Zhonghua Xi
Abstract:
Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or…
▽ More
Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio- and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on task-specific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos, with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech co-occurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.
△ Less
Submitted 23 August, 2018; v1 submitted 1 August, 2018;
originally announced August 2018.
