-
Audio-video fusion strategies for active speaker detection in meetings
Authors:
Lionel Pibre,
Francisco Madrigal,
Cyrille Equoy,
Frédéric Lerasle,
Thomas Pellegrini,
Julien Pinquier,
Isabelle Ferrané
Abstract:
Meetings are a common activity in professional contexts, and it remains challenging to endow vocal assistants with advanced functionalities to facilitate meeting management. In this context, a task like active speaker detection can provide useful insights to model interaction between meeting participants. Motivated by our application context related to advanced meeting assistant, we want to combin…
▽ More
Meetings are a common activity in professional contexts, and it remains challenging to endow vocal assistants with advanced functionalities to facilitate meeting management. In this context, a task like active speaker detection can provide useful insights to model interaction between meeting participants. Motivated by our application context related to advanced meeting assistant, we want to combine audio and visual information to achieve the best possible performance. In this paper, we propose two different types of fusion for the detection of the active speaker, combining two visual modalities and an audio modality through neural networks. For comparison purpose, classical unsupervised approaches for audio feature extraction are also used. We expect visual data centered on the face of each participant to be very appropriate for detecting voice activity, based on the detection of lip and facial gestures. Thus, our baseline system uses visual data and we chose a 3D Convolutional Neural Network architecture, which is effective for simultaneously encoding appearance and movement. To improve this system, we supplemented the visual information by processing the audio stream with a CNN or an unsupervised speaker diarization system. We have further improved this system by adding visual modality information using motion through optical flow. We evaluated our proposal with a public and state-of-the-art benchmark: the AMI corpus. We analysed the contribution of each system to the merger carried out in order to determine if a given participant is currently speaking. We also discussed the results we obtained. Besides, we have shown that, for our application context, adding motion information greatly improves performance. Finally, we have shown that attention-based fusion improves performance while reducing the standard deviation.
△ Less
Submitted 9 June, 2022;
originally announced June 2022.
-
Tackling the phase problem of diffraction for retrieval of photonic structures formed in nanocomposite materials
Authors:
M. Fally,
Y. Tomita,
A. Fimia,
R. F. Madrigal,
J. Guo,
J. Kohlbrecher,
J. Klepp
Abstract:
We experimentally demonstrate how to solve the phase problem of diffraction using multi-wave interference with standard diffraction experimental setups without the need for taking any auxiliary data. In particular, we show that the phases of the Fourier components of a periodic structure can be fully recovered by deliberately choosing a probe wavelength of the diffracting radiation much smaller th…
▽ More
We experimentally demonstrate how to solve the phase problem of diffraction using multi-wave interference with standard diffraction experimental setups without the need for taking any auxiliary data. In particular, we show that the phases of the Fourier components of a periodic structure can be fully recovered by deliberately choosing a probe wavelength of the diffracting radiation much smaller than the lattice constant. In the course of the demonstration, we accurately determine the refractive index profile of nanocomposite phase gratings by light and neutron diffraction measurements.
△ Less
Submitted 13 November, 2020;
originally announced November 2020.
-
Internal structure of superclusters of galaxies from pattern recognition techniques
Authors:
I. Santiago-Bautista,
C. A. Caretta,
H. Bravo-Alfaro,
E. Pointecouteau,
F. Madrigal
Abstract:
The Large-Scale Structure (LSS) of the Universe is a homogeneous network of galaxies separated in dense complexes, the superclusters of galaxies, and almost empty voids. The superclusters are young structures that did not have time to evolve into dynamically relaxed systems through the age of the Universe. Internally, they are very irregular, with dense cores, filaments and peripheral systems of g…
▽ More
The Large-Scale Structure (LSS) of the Universe is a homogeneous network of galaxies separated in dense complexes, the superclusters of galaxies, and almost empty voids. The superclusters are young structures that did not have time to evolve into dynamically relaxed systems through the age of the Universe. Internally, they are very irregular, with dense cores, filaments and peripheral systems of galaxies. We propose a methodology to map the internal structure of superclusters of galaxies using pattern recognition techniques. Our approach allows to: i) identify groups and clusters in the LSS distribution of galaxies; ii) correct for the "fingers of God" projection effect, caused by the partial knowledge of the third space coordinate; iii) detect filaments of galaxies and trace their skeletons. In this paper, we present the algorithms, discuss the optimization of the free parameters and evaluate the results of its application. With this methodology, we have mapped the internal structure of 42 superclusters in the nearby universe (up to $z=0.15$).
△ Less
Submitted 9 January, 2020;
originally announced January 2020.
-
LinTO : Assistant vocal open-source respectueux des données personnelles pour les réunions d'entreprise
Authors:
Jean-Pierre Lorré,
Isabelle Ferrané,
Francisco Madrigal,
Michalis Vazirgiannis,
Christophe Bourguignat
Abstract:
This paper presents the first results of the PIA "Grands Défis du Numérique" research project LinTO. The goal of this project is to develop a conversational assistant to help the company's employees, particularly during meetings. LinTO is an interactive device equipped with microphones, a screen and a 360$^\circ$ camera, which allows to control the room, query company's information system, helps f…
▽ More
This paper presents the first results of the PIA "Grands Défis du Numérique" research project LinTO. The goal of this project is to develop a conversational assistant to help the company's employees, particularly during meetings. LinTO is an interactive device equipped with microphones, a screen and a 360$^\circ$ camera, which allows to control the room, query company's information system, helps facilitate the meeting and provides an environment to aid minute writing. Distributed according to an open model that respects private data LinTO is the first open-source enterprise's assistant designed to comply with the GDPR requirements.
△ Less
Submitted 17 July, 2019;
originally announced July 2019.
