-
A multimodal LLM for the non-invasive decoding of spoken text from brain recordings
Authors:
Youssef Hmamouche,
Ismail Chihab,
Lahoucine Kdouri,
Amal El Fallah Seghrouchni
Abstract:
Brain-related research topics in artificial intelligence have recently gained popularity, particularly due to the expansion of what multimodal architectures can do from computer vision to natural language processing. Our main goal in this work is to explore the possibilities and limitations of these architectures in spoken text decoding from non-invasive fMRI recordings. Contrary to vision and tex…
▽ More
Brain-related research topics in artificial intelligence have recently gained popularity, particularly due to the expansion of what multimodal architectures can do from computer vision to natural language processing. Our main goal in this work is to explore the possibilities and limitations of these architectures in spoken text decoding from non-invasive fMRI recordings. Contrary to vision and textual data, fMRI data represent a complex modality due to the variety of brain scanners, which implies (i) the variety of the recorded signal formats, (ii) the low resolution and noise of the raw signals, and (iii) the scarcity of pretrained models that can be leveraged as foundation models for generative learning. These points make the problem of the non-invasive decoding of text from fMRI recordings very challenging. In this paper, we propose and end-to-end multimodal LLM for decoding spoken text from fMRI signals. The proposed architecture is founded on (i) an encoder derived from a specific transformer incorporating an augmented embedding layer for the encoder and a better-adjusted attention mechanism than that present in the state of the art, and (ii) a frozen large language model adapted to align the embedding of the input text and the encoded embedding of brain activity to decode the output text. A benchmark in performed on a corpus consisting of a set of interactions human-human and human-robot interactions where fMRI and conversational signals are recorded synchronously. The obtained results are very promising, as our proposal outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth. The implementation code is provided in https://github.com/Hmamouche/brain_decode.
△ Less
Submitted 29 September, 2024;
originally announced September 2024.
-
Link Budget Analysis for Reconfigurable Smart Surfaces in Aerial Platforms
Authors:
Safwan Alfattani,
Wael Jaafar,
Yassine Hmamouche,
Halim Yanikomeroglu,
Abbas Yongaçoglu
Abstract:
Non-terrestrial networks, including Unmanned Aerial Vehicles (UAVs), High Altitude Platform Station (HAPS) and Low Earth Orbiting (LEO) satellites, are expected to have a pivotal role in the sixth generation wireless networks. With their inherent features such as flexible placement, wide footprint, and preferred channel conditions, they can tackle several challenges in current terrestrial networks…
▽ More
Non-terrestrial networks, including Unmanned Aerial Vehicles (UAVs), High Altitude Platform Station (HAPS) and Low Earth Orbiting (LEO) satellites, are expected to have a pivotal role in the sixth generation wireless networks. With their inherent features such as flexible placement, wide footprint, and preferred channel conditions, they can tackle several challenges in current terrestrial networks However, their successful and widespread adoption relies on energy-efficient on-board communication systems. In this context, the integration of Reconfigurable Smart Surfaces (RSS) into aerial platforms is envisioned as a key enabler of energy-efficient and cost-effective deployments of aerial platforms. Indeed, RSS consist of low-cost reflectors capable of smartly directing signals in a nearly passive way. We investigate in this paper the link budget of RSS-assisted communications under the two discussed RSS reflection paradigms in the literature, namely the specular and the scattering reflection paradigm types. Specifically, we analyze the characteristics of RSS-equipped aerial platforms and compare their communication performance with that of RSS-assisted terrestrial networks, using standardized channel models. In addition, we derive the optimal aerial platforms placements under both reflection paradigms. The obtained results provide important insights for the design of RSS-assisted communications. For instance, given that a HAPS has a large RSS surface, it provides superior link budget performance in most studied scenarios. In contrast, the limited RSS area on UAVs and the large propagation loss in LEO satellite communications make them unfavorable candidates for supporting terrestrial users. Finally, the optimal location of the RSS-equipped platform may depend on the platform's altitude, coverage footprint, and type of environment.
△ Less
Submitted 16 August, 2021; v1 submitted 27 August, 2020;
originally announced August 2020.
-
Aerial Platforms with Reconfigurable Smart Surfaces for 5G and Beyond
Authors:
Safwan Alfattani,
Wael Jaafar,
Yassine Hmamouche,
Halim Yanikomeroglu,
Abbas Yongaçoglu,
Ng\d{o}c Dũng Đào,
Peiying Zhu
Abstract:
Aerial platforms are expected to deliver enhanced and seamless connectivity in the fifth generation (5G) wireless networks and beyond (B5G). This is generally achievable by supporting advanced onboard communication features embedded in heavy and energy-intensive equipment. Alternatively, reconfigurable smart surfaces (RSS), which smartly exploit/recycle signal reflections in the environment, are i…
▽ More
Aerial platforms are expected to deliver enhanced and seamless connectivity in the fifth generation (5G) wireless networks and beyond (B5G). This is generally achievable by supporting advanced onboard communication features embedded in heavy and energy-intensive equipment. Alternatively, reconfigurable smart surfaces (RSS), which smartly exploit/recycle signal reflections in the environment, are increasingly being recognized as a new wireless communication paradigm to improve communication links. In fact, their reduced cost, low power use, light weight, and flexible deployment make them an attractive candidate for integration with 5G/B5G technologies. In this article, we discuss comprehensive approaches to the integration of RSS in aerial platforms. First, we present a review of RSS technology, its operations and types of communication. Next, we describe how RSS can be used in aerial platforms, and we propose a control architecture workflow. Then, several potential use cases are presented and discussed. Finally, associated research challenges are identified.
△ Less
Submitted 4 November, 2020; v1 submitted 16 June, 2020;
originally announced June 2020.
