NWT: Towards natural audio-to-video generation with representation learning
Authors:
Rayhane Mama,
Marc S. Tyndel,
Hashiam Kadhim,
Cole Clifford,
Ragavan Thurairatnam
Abstract:
In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent…
▽ More
In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO's Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness, facial naturalness and expressiveness, and lipsync quality. This work sets a strong baseline for generalized audio-to-video synthesis. Samples are available at https://next-week-tonight.github.io/NWT/.
△ Less
Submitted 8 June, 2021;
originally announced June 2021.
Combining multi-site Magnetic Resonance Imaging with machine learning predicts survival in paediatric brain tumours
Authors:
James T. Grist,
Stephanie Withey,
Christopher Bennett,
Heather E. L. Rose,
Lesley MacPherson,
Adam Oates,
Stephen Powell,
Jan Novak,
Laurence Abernethy,
Barry Pizer,
Simon Bailey,
Steven C. Clifford,
Dipayan Mitra,
Theodoros N. Arvanitis,
Dorothee P. Auer,
Shivaram Avula,
Richard Grundy,
Andrew C Peet
Abstract:
Background Brain tumours represent the highest cause of mortality in the paediatric oncological population. Diagnosis is commonly performed with magnetic resonance imaging and spectroscopy. Survival biomarkers are challenging to identify due to the relatively low numbers of individual tumour types, especially for rare tumour types such as atypical rhabdoid tumours.
Methods 69 children with biops…
▽ More
Background Brain tumours represent the highest cause of mortality in the paediatric oncological population. Diagnosis is commonly performed with magnetic resonance imaging and spectroscopy. Survival biomarkers are challenging to identify due to the relatively low numbers of individual tumour types, especially for rare tumour types such as atypical rhabdoid tumours.
Methods 69 children with biopsy-confirmed brain tumours were recruited into this study. All participants had both perfusion and diffusion weighted imaging performed at diagnosis. Data were processed using conventional methods, and a Bayesian survival analysis performed. Unsupervised and supervised machine learning were performed with the survival features, to determine novel sub-groups related to survival. Sub-group analysis was undertaken to understand differences in imaging features, which pertain to survival.
Findings Survival analysis showed that a combination of diffusion and perfusion imaging were able to determine two novel sub-groups of brain tumours with different survival characteristics (p <0.01), which were subsequently classified with high accuracy (98%) by a neural network. Further analysis of high-grade tumours showed a marked difference in survival (p=0.029) between the two clusters with high risk and low risk imaging features.
Interpretation This study has developed a novel model of survival for paediatric brain tumours, with an implementation ready for integration into clinical practice. Results show that tumour perfusion plays a key role in determining survival in brain tumours and should be considered as a high priority for future imaging protocols.
△ Less
Submitted 21 April, 2020;
originally announced April 2020.
