Showing 1–50 of 77 results for author: Luo, H

An Improved Finite Element Modeling Method for Triply Periodic Minimal Surface Structures Based on Element Size and Minimum Jacobian

Authors: Siqi Wang, Chuangyu Jiang, Xiaodong Zhang, Yilong Zhang, Baoqiang Zhang, Huageng Luo

Abstract: Triply periodic minimal surface (TPMS) structures, a type of lattice structure, have garnered significant attention due to their lightweight nature, controllability, and excellent mechanical properties. Voxel-based modeling is a widely used method for investigating the mechanical behavior of such lattice structures through finite element simulations. This study proposes a two-parameter voxel metho… ▽ More Triply periodic minimal surface (TPMS) structures, a type of lattice structure, have garnered significant attention due to their lightweight nature, controllability, and excellent mechanical properties. Voxel-based modeling is a widely used method for investigating the mechanical behavior of such lattice structures through finite element simulations. This study proposes a two-parameter voxel method that incorporates joint control of element size and minimum Jacobian (MJ). Numerical results indicate that the simulation outcomes tend to stabilize when the MJ reaches 0.3. The grid convergence index (GCI), based on Richardson extrapolation, is introduced to systematically assess the numerical convergence behavior of both voxel models and the proposed two-parameter voxel models. This provides a systematic and objective framework for evaluating discretization errors and mesh convergence in TPMS modeling. Compared with traditional voxel method, the proposed method exhibits superior mesh convergence, solution accuracy, and computational efficiency. Furthermore, the two-parameter voxel method also shows excellent applicability in the analysis of graded TPMS structures, exhibiting even better convergence behavior than in uniform structures. △ Less

Submitted 4 June, 2025; originally announced June 2025.

Dual-Polarization Stacked Intelligent Metasurfaces for Holographic MIMO

Authors: Yida Zhang, Qiuyan Liu, Hongtao Luo, Yuqi Xia, Qiang Wang

Abstract: To address the limited wave domain signal processing capabilities of traditional single-polarized stacked intelligent metasurfaces (SIMs) in holographic multiple-input multiple-output (HMIMO) systems, which stems from limited integration space, this paper proposes a dual-polarized SIM (DPSIM) architecture. By stacking dual-polarized reconfigurable intelligent surfaces (DPRIS), DPSIM can independen… ▽ More To address the limited wave domain signal processing capabilities of traditional single-polarized stacked intelligent metasurfaces (SIMs) in holographic multiple-input multiple-output (HMIMO) systems, which stems from limited integration space, this paper proposes a dual-polarized SIM (DPSIM) architecture. By stacking dual-polarized reconfigurable intelligent surfaces (DPRIS), DPSIM can independently process signals of two orthogonal polarizations in the wave domain, thereby effectively suppressing polarization cross-interference (PCI) and inter-stream interference (ISI). We introduce a layer-by-layer gradient descent with water-filling (LGD-WF) algorithm to enhance end-to-end performance. Simulation results show that, under the same number of metasurface layers and unit size, the DPSIM-aided HMIMO system can support more simultaneous data streams for ISI-free parallel transmission compared to traditional SIM-aided systems. Furthermore, under different polarization imperfection conditions, both the spectral efficiency (SE) and energy efficiency (EE) of the DPSIM-aided HMIMO system are significantly improved, approaching the theoretical upper bound. △ Less

Submitted 27 May, 2025; originally announced May 2025.

AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models

Authors: Kai Li, Can Shen, Yile Liu, Jirui Han, Kelong Zheng, Xuechao Zou, Zhe Wang, Xingjian Du, Shun Zhang, Hanjun Luo, Yingbin Jin, Xinxin Xing, Ziyang Ma, Yue Liu, Xiaojun Jia, Yifan Zhang, Junfeng Fang, Kun Wang, Yibo Yan, Haoyang Li, Yiming Li, Xiaobin Zhuang, Yang Liu, Haibo Hu, Zhizheng Wu , et al. (6 additional authors not shown)

Abstract: The rapid advancement and expanding applications of Audio Large Language Models (ALLMs) demand a rigorous understanding of their trustworthiness. However, systematic research on evaluating these models, particularly concerning risks unique to the audio modality, remains largely unexplored. Existing evaluation frameworks primarily focus on the text modality or address only a restricted set of safet… ▽ More The rapid advancement and expanding applications of Audio Large Language Models (ALLMs) demand a rigorous understanding of their trustworthiness. However, systematic research on evaluating these models, particularly concerning risks unique to the audio modality, remains largely unexplored. Existing evaluation frameworks primarily focus on the text modality or address only a restricted set of safety dimensions, failing to adequately account for the unique characteristics and application scenarios inherent to the audio modality. We introduce AudioTrust-the first multifaceted trustworthiness evaluation framework and benchmark specifically designed for ALLMs. AudioTrust facilitates assessments across six key dimensions: fairness, hallucination, safety, privacy, robustness, and authentication. To comprehensively evaluate these dimensions, AudioTrust is structured around 18 distinct experimental setups. Its core is a meticulously constructed dataset of over 4,420 audio/text samples, drawn from real-world scenarios (e.g., daily conversations, emergency calls, voice assistant interactions), specifically designed to probe the multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully designs 9 audio-specific evaluation metrics, and we employ a large-scale automated pipeline for objective and scalable scoring of model outputs. Experimental results reveal the trustworthiness boundaries and limitations of current state-of-the-art open-source and closed-source ALLMs when confronted with various high-risk audio scenarios, offering valuable insights for the secure and trustworthy deployment of future audio models. Our platform and benchmark are available at https://github.com/JusperLee/AudioTrust. △ Less

Submitted 1 July, 2025; v1 submitted 22 May, 2025; originally announced May 2025.

Comments: Technical Report

Sensing Framework Design and Performance Optimization with Action Detection for ISCC

Authors: Weiwei Chen, Yinghui He, Guanding Yu, Jianfeng Wang, Haiyan Luo

Abstract: Integrated sensing, communication, and computation (ISCC) has been regarded as a prospective technology for the next-generation wireless network, supporting humancentric intelligent applications. However, the delay sensitivity of these computation-intensive applications, especially in a multidevice ISCC system with limited resources, highlights the urgent need for efficient sensing task execution… ▽ More Integrated sensing, communication, and computation (ISCC) has been regarded as a prospective technology for the next-generation wireless network, supporting humancentric intelligent applications. However, the delay sensitivity of these computation-intensive applications, especially in a multidevice ISCC system with limited resources, highlights the urgent need for efficient sensing task execution frameworks. To address this, we propose a resource-efficient sensing framework in this paper. Different from existing solutions, it features a novel action detection module deployed at each device to detect the onset of an action. Only time windows filled with signals of interest are offloaded to the edge server and processed by the edge recognition module, thus reducing overhead. Furthermore, we quantitatively analyze the sensing performance of the proposed sensing framework and formulate a sensing accuracy maximization problem under power, delay, and resource limitations for the multi-device ISCC system. By decomposing it into two subproblems, we develop an alternating direction method of multipliers (ADMM)-based distributed algorithm. It alternatively solves a sensing accuracy maximization subproblem at each device and employs a closed-form computation resource allocation strategy at the edge server till convergence. Finally, a real-world test is conducted using commodity wireless devices to validate the sensing performance analysis. Extensive test results demonstrate that our proposal achieves higher sensing accuracy under the limited resource compared to two baselines. △ Less

Submitted 5 May, 2025; originally announced May 2025.

Comments: Accepted by IEEE Transactions on Wireless Communications

Sphere Precoding for Robust Near-Field Communications

Authors: Hao Luo, Yu Zhang, Ahmed Alkhateeb

Abstract: Near-field communication with large antenna arrays promises significant beamforming and multiplexing gains. These communication links, however, are very sensitive to user mobility as any small change in the user position may suddenly drop the signal power. This leads to critical challenges for the robustness of these near-field communication systems. In this paper, we propose \textit{sphere precod… ▽ More Near-field communication with large antenna arrays promises significant beamforming and multiplexing gains. These communication links, however, are very sensitive to user mobility as any small change in the user position may suddenly drop the signal power. This leads to critical challenges for the robustness of these near-field communication systems. In this paper, we propose \textit{sphere precoding}, which is a robust precoding design to address user mobility in near-field communications. To gain insights into the spatial correlation of near-field channels, we extend the one-ring channel model to what we call one-sphere channel model and derive the channel covariance considering user mobility. Based on the one-sphere channel model, a robust precoding design problem is defined to optimize the minimum signal-to-interference-plus-noise ratio (SINR) satisfaction probability among mobile users. By utilizing the eigen structure of channel covariance, we further design a relaxed convex problem to approximate the solution of the original non-convex problem. The low-complexity solution effectively shapes a sphere that maintains the signal power for the target user and also nulls its interference within spheres around the other users. Simulation results highlight the efficacy of the proposed solution in achieving robust precoding yet high achievable rates in near-field communication systems. △ Less

Submitted 9 March, 2025; originally announced March 2025.

Comments: The code for sphere precoding will be available on the Wireless Intelligence Lab website: https://www.wi-lab.net/

Multi-illuminant Color Constancy via Multi-scale Illuminant Estimation and Fusion

Authors: Hang Luo, Rongwei Li, Jinxing Liang

Abstract: Multi-illuminant color constancy methods aim to eliminate local color casts within an image through pixel-wise illuminant estimation. Existing methods mainly employ deep learning to establish a direct mapping between an image and its illumination map, which neglects the impact of image scales. To alleviate this problem, we represent an illuminant map as the linear combination of components estimat… ▽ More Multi-illuminant color constancy methods aim to eliminate local color casts within an image through pixel-wise illuminant estimation. Existing methods mainly employ deep learning to establish a direct mapping between an image and its illumination map, which neglects the impact of image scales. To alleviate this problem, we represent an illuminant map as the linear combination of components estimated from multi-scale images. Furthermore, we propose a tri-branch convolution networks to estimate multi-grained illuminant distribution maps from multi-scale images. These multi-grained illuminant maps are merged adaptively with an attentional illuminant fusion module. Through comprehensive experimental analysis and evaluation, the results demonstrate the effectiveness of our method, and it has achieved state-of-the-art performance. △ Less

Submitted 4 February, 2025; originally announced February 2025.

Comments: 10 pages, 4 figures, this manuscript is under the consideration of Optics Express

MinMo: A Multimodal Large Language Model for Seamless Voice Interaction

Authors: Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan , et al. (11 additional authors not shown)

Abstract: Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence le… ▽ More Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo, and the code and models will be released soon. △ Less

Submitted 10 January, 2025; originally announced January 2025.

Comments: Work in progress. Authors are listed in alphabetical order by family name

A Paragraph is All It Takes: Rich Robot Behaviors from Interacting, Trusted LLMs

Authors: OpenMind, Shaohong Zhong, Adam Zhou, Boyuan Chen, Homin Luo, Jan Liphardt

Abstract: Large Language Models (LLMs) are compact representations of all public knowledge of our physical environment and animal and human behaviors. The application of LLMs to robotics may offer a path to highly capable robots that perform well across most human tasks with limited or even zero tuning. Aside from increasingly sophisticated reasoning and task planning, networks of (suitably designed) LLMs o… ▽ More Large Language Models (LLMs) are compact representations of all public knowledge of our physical environment and animal and human behaviors. The application of LLMs to robotics may offer a path to highly capable robots that perform well across most human tasks with limited or even zero tuning. Aside from increasingly sophisticated reasoning and task planning, networks of (suitably designed) LLMs offer ease of upgrading capabilities and allow humans to directly observe the robot's thinking. Here we explore the advantages, limitations, and particularities of using LLMs to control physical robots. The basic system consists of four LLMs communicating via a human language data bus implemented via web sockets and ROS2 message passing. Surprisingly, rich robot behaviors and good performance across different tasks could be achieved despite the robot's data fusion cycle running at only 1Hz and the central data bus running at the extremely limited rates of the human brain, of around 40 bits/s. The use of natural language for inter-LLM communication allowed the robot's reasoning and decision making to be directly observed by humans and made it trivial to bias the system's behavior with sets of rules written in plain English. These rules were immutably written into Ethereum, a global, public, and censorship resistant Turing-complete computer. We suggest that by using natural language as the data bus among interacting AIs, and immutable public ledgers to store behavior constraints, it is possible to build robots that combine unexpectedly rich performance, upgradability, and durable alignment with humans. △ Less

Submitted 24 December, 2024; originally announced December 2024.

Comments: 10 pages, 1 figure

User Identity Protection in EEG-based Brain-Computer Interfaces

Authors: L. Meng, X. Jiang, J. Huang, W. Li, H. Luo, D. Wu

Abstract: A brain-computer interface (BCI) establishes a direct communication pathway between the brain and an external device. Electroencephalogram (EEG) is the most popular input signal in BCIs, due to its convenience and low cost. Most research on EEG-based BCIs focuses on the accurate decoding of EEG signals; however, EEG signals also contain rich private information, e.g., user identity, emotion, and s… ▽ More A brain-computer interface (BCI) establishes a direct communication pathway between the brain and an external device. Electroencephalogram (EEG) is the most popular input signal in BCIs, due to its convenience and low cost. Most research on EEG-based BCIs focuses on the accurate decoding of EEG signals; however, EEG signals also contain rich private information, e.g., user identity, emotion, and so on, which should be protected. This paper first exposes a serious privacy problem in EEG-based BCIs, i.e., the user identity in EEG data can be easily learned so that different sessions of EEG data from the same user can be associated together to more reliably mine private information. To address this issue, we further propose two approaches to convert the original EEG data into identity-unlearnable EEG data, i.e., removing the user identity information while maintaining the good performance on the primary BCI task. Experiments on seven EEG datasets from five different BCI paradigms showed that on average the generated identity-unlearnable EEG data can reduce the user identification accuracy from 70.01\% to at most 21.36\%, greatly facilitating user privacy protection in EEG-based BCIs. △ Less

Submitted 12 December, 2024; originally announced December 2024.

Journal ref: IEEE Trans. on Neural Systems and Rehabilitation Engineering, 31:3576-3586, 2023

Efficient Satellite-Ground Interconnection Design for Low-orbit Mega-Constellation Topology

Authors: Wenhao Liu, Jiazhi Wu, Quanwei Lin, Handong Luo, Qi Zhang, Kun Qiu, Zhe Chen, Yue Gao

Abstract: The low-orbit mega-constellation network (LMCN) is an important part of the space-air-ground integrated network system. An effective satellite-ground interconnection design can result in a stable constellation topology for LMCNs. A naive solution is accessing the satellite with the longest remaining service time (LRST), which is widely used in previous designs. The Coordinated Satellite-Ground Int… ▽ More The low-orbit mega-constellation network (LMCN) is an important part of the space-air-ground integrated network system. An effective satellite-ground interconnection design can result in a stable constellation topology for LMCNs. A naive solution is accessing the satellite with the longest remaining service time (LRST), which is widely used in previous designs. The Coordinated Satellite-Ground Interconnecting (CSGI), the state-of-the-art algorithm, coordinates the establishment of ground-satellite links (GSLs). Compared with existing solutions, it reduces latency by 19% and jitter by 70% on average. However, CSGI only supports the scenario where terminals access only one satellite and cannot fully utilize the multi-access capabilities of terminals. Additionally, CSGI's high computational complexity poses deployment challenges. To overcome these problems, we propose the Classification-based Longest Remaining Service Time (C-LRST) algorithm. C-LRST supports the actual scenario with multi-access capabilities. It adds optional paths during routing with low computational complexity, improving end-to-end communications quality. We conduct our 1000s simulation from Brazil to Lithuania on the open-source platform Hypatia. Experiment results show that compared with CSGI, C-LRST reduces the latency and increases the throughput by approximately 60% and 40%, respectively. In addition, C-LRST's GSL switching number is 14, whereas CSGI is 23. C-LRST has better link stability than CSGI. △ Less

Submitted 31 October, 2024; originally announced October 2024.

Comments: 13 pages, 14 figures

Hierarchical Attention and Parallel Filter Fusion Network for Multi-Source Data Classification

Authors: Han Luo, Feng Gao, Junyu Dong, Lin Qi

Abstract: Hyperspectral image (HSI) and synthetic aperture radar (SAR) data joint classification is a crucial and yet challenging task in the field of remote sensing image interpretation. However, feature modeling in existing methods is deficient to exploit the abundant global, spectral, and local features simultaneously, leading to sub-optimal classification performance. To solve the problem, we propose a… ▽ More Hyperspectral image (HSI) and synthetic aperture radar (SAR) data joint classification is a crucial and yet challenging task in the field of remote sensing image interpretation. However, feature modeling in existing methods is deficient to exploit the abundant global, spectral, and local features simultaneously, leading to sub-optimal classification performance. To solve the problem, we propose a hierarchical attention and parallel filter fusion network for multi-source data classification. Concretely, we design a hierarchical attention module for hyperspectral feature extraction. This module integrates global, spectral, and local features simultaneously to provide more comprehensive feature representation. In addition, we develop parallel filter fusion module which enhances cross-modal feature interactions among different spatial locations in the frequency domain. Extensive experiments on two multi-source remote sensing data classification datasets verify the superiority of our proposed method over current state-of-the-art classification approaches. Specifically, our proposed method achieves 91.44% and 80.51% of overall accuracy (OA) on the respective datasets, highlighting its superior performance. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: Accepted by IEEE GRSL

Convergence of Symbiotic Communications and Blockchain for Sustainable and Trustworthy 6G Wireless Networks

Authors: Haoxiang Luo, Gang Sun, Cheng Chi, Hongfang Yu, Mohsen Guizani

Abstract: Symbiotic communication (SC) is known as a new wireless communication paradigm, similar to the natural ecosystem population, and can enable multiple communication systems to cooperate and mutualize through service exchange and resource sharing. As a result, SC is seen as an important potential technology for future sixth-generation (6G) communications, solving the problem of lack of spectrum resou… ▽ More Symbiotic communication (SC) is known as a new wireless communication paradigm, similar to the natural ecosystem population, and can enable multiple communication systems to cooperate and mutualize through service exchange and resource sharing. As a result, SC is seen as an important potential technology for future sixth-generation (6G) communications, solving the problem of lack of spectrum resources and energy inefficiency. Symbiotic relationships among communication systems can complement radio resources in 6G. However, the absence of established trust relationships among diverse communication systems presents a formidable hurdle in ensuring efficient and trusted resource and service exchange within SC frameworks. To better realize trusted SC services in 6G, in this paper, we propose a solution that converges SC and blockchain, called a symbiotic blockchain network (SBN). Specifically, we first use cognitive backscatter communication to transform blockchain consensus, that is, the symbiotic blockchain consensus (SBC), so that it can be better suited for the wireless network. Then, for SBC, we propose a highly energy-efficient sharding scheme to meet the extremely low power consumption requirements in 6G. Finally, such a blockchain scheme guarantees trusted transactions of communication services in SC. Through ablation experiments, our proposed SBN demonstrates significant efficacy in mitigating energy consumption and reducing processing latency in adversarial networks, which is expected to achieve a sustainable and trusted 6G wireless network. △ Less

Submitted 11 August, 2024; originally announced August 2024.

Power-LLaVA: Large Language and Vision Assistant for Power Transmission Line Inspection

Authors: Jiahao Wang, Mingxuan Li, Haichen Luo, Jinguo Zhu, Aijun Yang, Mingzhe Rong, Xiaohua Wang

Abstract: The inspection of power transmission line has achieved notable achievements in the past few years, primarily due to the integration of deep learning technology. However, current inspection approaches continue to encounter difficulties in generalization and intelligence, which restricts their further applicability. In this paper, we introduce Power-LLaVA, the first large language and vision assista… ▽ More The inspection of power transmission line has achieved notable achievements in the past few years, primarily due to the integration of deep learning technology. However, current inspection approaches continue to encounter difficulties in generalization and intelligence, which restricts their further applicability. In this paper, we introduce Power-LLaVA, the first large language and vision assistant designed to offer professional and reliable inspection services for power transmission line by engaging in dialogues with humans. Moreover, we also construct a large-scale and high-quality dataset specialized for the inspection task. By employing a two-stage training strategy on the constructed dataset, Power-LLaVA demonstrates exceptional performance at a comparatively low training cost. Extensive experiments further prove the great capabilities of Power-LLaVA within the realm of power transmission line inspection. Code shall be released. △ Less

Submitted 27 July, 2024; originally announced July 2024.

6D Motion Parameters Estimation in Monostatic Integrated Sensing and Communications System

Authors: Hongliang Luo, Feifei Gao, Fan Liu, Shi Jin

Abstract: In this paper, we propose a novel scheme to estimate the six dimensional (6D) motion parameters of dynamic target for monostatic integrated sensing and communications (ISAC) system. We first provide a generic ISAC framework for dynamic target sensing based on massive multiple input and multiple output (MIMO) array. Next, we derive the relationship between the sensing channel of ISAC base station (… ▽ More In this paper, we propose a novel scheme to estimate the six dimensional (6D) motion parameters of dynamic target for monostatic integrated sensing and communications (ISAC) system. We first provide a generic ISAC framework for dynamic target sensing based on massive multiple input and multiple output (MIMO) array. Next, we derive the relationship between the sensing channel of ISAC base station (BS) and the 6D motion parameters of dynamic target. Then, we employ the array signal processing methods to estimate the horizontal angle, pitch angle, distance, and virtual velocity of dynamic target. Since the virtual velocities observed by different antennas are different, we adopt plane fitting to estimate the dynamic target's radial velocity, horizontal angular velocity, and pitch angular velocity from these virtual velocities. Simulation results demonstrate the effectiveness of the proposed 6D motion parameters estimation scheme, which also confirms a new finding that one single BS with massive MIMO array is capable of estimating the horizontal angular velocity and pitch angular velocity of dynamic target. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: arXiv admin note: substantial text overlap with arXiv:2312.16441

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

Authors: Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, Changhe Song, Jiaqi Shi, Xian Shi, Hao Wang, Wen Wang, Yuxuan Wang , et al. (8 additional authors not shown)

Abstract: This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, sp… ▽ More This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM. △ Less

Submitted 10 July, 2024; v1 submitted 4 July, 2024; originally announced July 2024.

Comments: Work in progress. Authors are listed in alphabetical order by family name

Trustworthy Enhanced Multi-view Multi-modal Alzheimer's Disease Prediction with Brain-wide Imaging Transcriptomics Data

Authors: Shan Cong, Zhoujie Fan, Hongwei Liu, Yinghan Zhang, Xin Wang, Haoran Luo, Xiaohui Yao

Abstract: Brain transcriptomics provides insights into the molecular mechanisms by which the brain coordinates its functions and processes. However, existing multimodal methods for predicting Alzheimer's disease (AD) primarily rely on imaging and sometimes genetic data, often neglecting the transcriptomic basis of brain. Furthermore, while striving to integrate complementary information between modalities,… ▽ More Brain transcriptomics provides insights into the molecular mechanisms by which the brain coordinates its functions and processes. However, existing multimodal methods for predicting Alzheimer's disease (AD) primarily rely on imaging and sometimes genetic data, often neglecting the transcriptomic basis of brain. Furthermore, while striving to integrate complementary information between modalities, most studies overlook the informativeness disparities between modalities. Here, we propose TMM, a trusted multiview multimodal graph attention framework for AD diagnosis, using extensive brain-wide transcriptomics and imaging data. First, we construct view-specific brain regional co-function networks (RRIs) from transcriptomics and multimodal radiomics data to incorporate interaction information from both biomolecular and imaging perspectives. Next, we apply graph attention (GAT) processing to each RRI network to produce graph embeddings and employ cross-modal attention to fuse transcriptomics-derived embedding with each imagingderived embedding. Finally, a novel true-false-harmonized class probability (TFCP) strategy is designed to assess and adaptively adjust the prediction confidence of each modality for AD diagnosis. We evaluate TMM using the AHBA database with brain-wide transcriptomics data and the ADNI database with three imaging modalities (AV45-PET, FDG-PET, and VBM-MRI). The results demonstrate the superiority of our method in identifying AD, EMCI, and LMCI compared to state-of-the-arts. Code and data are available at https://github.com/Yaolab-fantastic/TMM. △ Less

Submitted 2 April, 2025; v1 submitted 21 June, 2024; originally announced June 2024.

Integrated Sensing and Communications Framework for 6G Networks

Authors: Hongliang Luo, Tengyu Zhang, Chuanbin Zhao, Yucong Wang, Bo Lin, Yuhua Jiang, Dongqi Luo, Feifei Gao

Abstract: In this paper, we propose a novel integrated sensing and communications (ISAC) framework for the sixth generation (6G) mobile networks, in which we decompose the real physical world into static environment, dynamic targets, and various object materials. The ubiquitous static environment occupies the vast majority of the physical world, for which we design static environment reconstruction (SER) sc… ▽ More In this paper, we propose a novel integrated sensing and communications (ISAC) framework for the sixth generation (6G) mobile networks, in which we decompose the real physical world into static environment, dynamic targets, and various object materials. The ubiquitous static environment occupies the vast majority of the physical world, for which we design static environment reconstruction (SER) scheme to obtain the layout and point cloud information of static buildings. The dynamic targets floating in static environments create the spatiotemporal transition of the physical world, for which we design comprehensive dynamic target sensing (DTS) scheme to detect, estimate, track, image and recognize the dynamic targets in real-time. The object materials enrich the electromagnetic laws of the physical world, for which we develop object material recognition (OMR) scheme to estimate the electromagnetic coefficient of the objects. Besides, to integrate these sensing functions into existing communications systems, we discuss the interference issues and corresponding solutions for ISAC cellular networks. Furthermore, we develop an ISAC hardware prototype platform that can reconstruct the environmental maps and sense the dynamic targets while maintaining communications services. With all these designs, the proposed ISAC framework can support multifarious emerging applications, such as digital twins, low altitude economy, internet of vehicles, marine management, deformation monitoring, etc. △ Less

Submitted 30 May, 2024; originally announced May 2024.

"Pass the butter": A study on desktop-classic multitasking robotic arm based on advanced YOLOv7 and BERT

Authors: Haohua Que, Wenbin Pan, Jie Xu, Hao Luo, Pei Wang, Li Zhang

Abstract: In recent years, various intelligent autonomous robots have begun to appear in daily life and production. Desktop-level robots are characterized by their flexible deployment, rapid response, and suitability for light workload environments. In order to meet the current societal demand for service robot technology, this study proposes using a miniaturized desktop-level robot (by ROS) as a carrier, l… ▽ More In recent years, various intelligent autonomous robots have begun to appear in daily life and production. Desktop-level robots are characterized by their flexible deployment, rapid response, and suitability for light workload environments. In order to meet the current societal demand for service robot technology, this study proposes using a miniaturized desktop-level robot (by ROS) as a carrier, locally deploying a natural language model (NLP-BERT), and integrating visual recognition (CV-YOLO) and speech recognition technology (ASR-Whisper) as inputs to achieve autonomous decision-making and rational action by the desktop robot. Three comprehensive experiments were designed to validate the robotic arm, and the results demonstrate excellent performance using this approach across all three experiments. In Task 1, the execution rates for speech recognition and action performance were 92.6% and 84.3%, respectively. In Task 2, the highest execution rates under the given conditions reached 92.1% and 84.6%, while in Task 3, the highest execution rates were 95.2% and 80.8%, respectively. Therefore, it can be concluded that the proposed solution integrating ASR, NLP, and other technologies on edge devices is feasible and provides a technical and engineering foundation for realizing multimodal desktop-level robots. △ Less

Submitted 27 May, 2024; originally announced May 2024.

Digital Twin Aided Compressive Sensing: Enabling Site-Specific MIMO Hybrid Precoding

Authors: Hao Luo, Ahmed Alkhateeb

Abstract: Compressive sensing is a promising solution for the channel estimation in multiple-input multiple-output (MIMO) systems with large antenna arrays and constrained hardware. Utilizing site-specific channel data from real-world systems, deep learning can be employed to learn the compressive sensing measurement vectors with minimum redundancy, thereby focusing sensing power on promising spatial direct… ▽ More Compressive sensing is a promising solution for the channel estimation in multiple-input multiple-output (MIMO) systems with large antenna arrays and constrained hardware. Utilizing site-specific channel data from real-world systems, deep learning can be employed to learn the compressive sensing measurement vectors with minimum redundancy, thereby focusing sensing power on promising spatial directions of the channel. Collecting real-world channel data, however, is challenging due to the high overhead resulting from the large number of antennas and hardware constraints. In this paper, we propose leveraging a site-specific digital twin to generate synthetic channel data, which shares a similar distribution with real-world data. The synthetic data is then used to train the deep learning models for learning measurement vectors and hybrid precoder/combiner design in an end-to-end manner. We further propose a model refinement approach to fine-tune the model pre-trained on the digital twin data with a small amount of real-world data. The evaluation results show that, by training the model on the digital twin data, the learned measurement vectors can be efficiently adapted to the environment geometry, leading to high performance of hybrid precoding for real-world deployments. Moreover, the model refinement approach can enable the digital twin aided model to achieve comparable performance to the model trained on the real-world dataset with a significantly reduced amount of real-world data. △ Less

Submitted 11 May, 2024; originally announced May 2024.

Comments: 7 pages, 5 figures

Joint Power Allocation and Beamforming for In-band Full-duplex Multi-cell Multi-user Networks

Authors: Haifeng Luo, Navneet Garg, Mark Holm, Tharmalingam Ratnarajah

Abstract: This paper investigates a robust joint power allocation and beamforming scheme for in-band full-duplex multi-cell multi-user (IBFD-MCMU) networks. A mean-squared error (MSE) minimization problem is formulated with constraints on the power budgets and residual self-interference (RSI) power. The problem is not convex, so we decompose it into two sub-problems: interference management beamforming and… ▽ More This paper investigates a robust joint power allocation and beamforming scheme for in-band full-duplex multi-cell multi-user (IBFD-MCMU) networks. A mean-squared error (MSE) minimization problem is formulated with constraints on the power budgets and residual self-interference (RSI) power. The problem is not convex, so we decompose it into two sub-problems: interference management beamforming and power allocation, and give closed-form solutions to the sub-problems. Then we propose an iterative algorithm to yield an overall solution. The computational complexity and convergence behavior of the algorithm are analyzed. Our method can enhance the analog self-interference cancellation (ASIC) depth provided by the precoder with less effect on the downlink communication than the existing null-space projection method, inspiring a low-cost but efficient IBFD transceiver design. It can achieve 42.9% of IBFD gain in terms of spectral efficiency with only antenna isolation, while this value increases to 60.9% with further digital self-interference cancellation (DSIC). Numerical results illustrate that our algorithm is robust to hardware impairments and channel uncertainty. With sufficient ASIC depth, our method reduces the computation time by at least 20% than the existing scheme due to its faster convergence speed at the cost of < 12.5% sum rate loss. The benefit is much more significant with single-antenna users that our algorithm saves at least 40% of the computation time at the cost of < 10% sum rate reduction. △ Less

Submitted 16 March, 2024; originally announced March 2024.

On the Secrecy Rate of In-Band Full-duplex Two-way Wiretap Channel

Authors: Navneet Garg, Haifeng Luo, Tharmalingam Ratnarajah

Abstract: In this paper, we consider a two-way wiretap Multi-Input Multi-Output Multi-antenna Eve (MIMOME) channel, where both nodes (Alice and Bob) transmit and receive in an in-band full-duplex (IBFD) manner. For this system with keyless security, we provide a novel artificial noise (AN) based signal design, where the AN is injected in both signal and null spaces. We present an ergodic secrecy rate approx… ▽ More In this paper, we consider a two-way wiretap Multi-Input Multi-Output Multi-antenna Eve (MIMOME) channel, where both nodes (Alice and Bob) transmit and receive in an in-band full-duplex (IBFD) manner. For this system with keyless security, we provide a novel artificial noise (AN) based signal design, where the AN is injected in both signal and null spaces. We present an ergodic secrecy rate approximation to derive the power allocation algorithm. We consider scenarios where AN is known and unknown to legitimate users and include imperfect channel information effects. To maximize secrecy rates subject to the transmit power constraint, a two-step power allocation solution is proposed, where the first step is known at Eve, and the second step helps to improve the secrecy further. We also consider scenarios where partial information is known by Eve and the effects of non-ideal self-interference cancellation. The usefulness and limitations of the resulting power allocation solution are analyzed and verified via simulations. Results show that secrecy rates are less when AN is unknown to receivers or Eve has more information about legitimate users. Since the ergodic approximation only considers Eves distance, the resulting power allocation provides secrecy rates close to the actual ones. △ Less

Submitted 11 March, 2024; originally announced March 2024.

Reinforcement Learning Based Robust Volt/Var Control in Active Distribution Networks With Imprecisely Known Delay

Authors: Hong Cheng, Huan Luo, Zhi Liu, Wei Sun, Weitao Li, Qiyue Li

Abstract: Active distribution networks (ADNs) incorporating massive photovoltaic (PV) devices encounter challenges of rapid voltage fluctuations and potential violations. Due to the fluctuation and intermittency of PV generation, the state gap, arising from time-inconsistent states and exacerbated by imprecisely known system delays, significantly impacts the accuracy of voltage control. This paper addresses… ▽ More Active distribution networks (ADNs) incorporating massive photovoltaic (PV) devices encounter challenges of rapid voltage fluctuations and potential violations. Due to the fluctuation and intermittency of PV generation, the state gap, arising from time-inconsistent states and exacerbated by imprecisely known system delays, significantly impacts the accuracy of voltage control. This paper addresses this challenge by introducing a framework for delay adaptive Volt/Var control (VVC) in the presence of imprecisely known system delays to regulate the reactive power of PV inverters. The proposed approach formulates the voltage control, based on predicted system operation states, as a robust VVC problem. It employs sample selection from the state prediction interval to promptly identify the worst-performing system operation state. Furthermore, we leverage the decentralized partially observable Markov decision process (Dec-POMDP) to reformulate the robust VVC problem. We design Multiple Policy Networks and employ Multiple Policy Networks and Reward Shaping-based Multi-agent Twin Delayed Deep Deterministic Policy Gradient (MPNRS-MATD3) algorithm to efficiently address and solve the Dec-POMDP model-based problem. Simulation results show the delay adaption characteristic of our proposed framework, and the MPNRS-MATD3 outperforms other multi-agent reinforcement learning algorithms in robust voltage control. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Integrated Imaging and Communication with Reconfigurable Intelligent Surfaces

Authors: Hao Luo, Ahmed Alkhateeb

Abstract: Reconfigurable intelligent surfaces, with their large number of antennas, offer an interesting opportunity for high spatial-resolution imaging. In this paper, we propose a novel RIS-aided integrated imaging and communication system that can reduce the RIS beam training overhead for communication by leveraging the imaging of the surrounding environment. In particular, using the RIS as a wireless im… ▽ More Reconfigurable intelligent surfaces, with their large number of antennas, offer an interesting opportunity for high spatial-resolution imaging. In this paper, we propose a novel RIS-aided integrated imaging and communication system that can reduce the RIS beam training overhead for communication by leveraging the imaging of the surrounding environment. In particular, using the RIS as a wireless imaging device, our system constructs the scene depth map of the environment, including the mobile user. Then, we develop a user detection algorithm that subtracts the background and extracts the mobile user attributes from the depth map. These attributes are then utilized to design the RIS interaction vector and the beam selection strategy with low overhead. Simulation results show that the proposed approach can achieve comparable beamforming gain to the optimal/exhaustive beam selection solution while requiring 1000 times less beam training overhead. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: 6 pages, 4 figures. To appear in Asilomar 2023

ISAC with Backscattering RFID Tags: Joint Beamforming Design

Authors: Hao Luo, Umut Demirhan, Ahmed Alkhateeb

Abstract: In this paper, we explore an integrated sensing and communication (ISAC) system with backscattering RFID tags. In this setup, an access point employs a communication beam to serve a user while leveraging a sensing beam to detect an RFID tag. Under the total transmit power constraint of the system, our objective is to design sensing and communication beams by considering the tag detection and commu… ▽ More In this paper, we explore an integrated sensing and communication (ISAC) system with backscattering RFID tags. In this setup, an access point employs a communication beam to serve a user while leveraging a sensing beam to detect an RFID tag. Under the total transmit power constraint of the system, our objective is to design sensing and communication beams by considering the tag detection and communication requirements. First, we adopt zero-forcing to design the beamforming vectors, followed by solving a convex optimization problem to determine the power allocation between sensing and communication. Then, we study a joint beamforming design problem with the goal of minimizing the total transmit power while satisfying the tag detection and communication requirements. To resolve this, we re-formulate the non-convex constraints into convex second-order cone constraints. The simulation results demonstrate that, under different communication SINR requirements, joint beamforming optimization outperforms the zero-forcing-based method in terms of achievable detection distance, offering a promising approach for the ISAC-backscattering systems. △ Less

Submitted 31 January, 2024; v1 submitted 18 January, 2024; originally announced January 2024.

Comments: 5 pages, 5 figures. To appear in IEEE ICC 2024

6D Radar Sensing and Tracking in Monostatic Integrated Sensing and Communications System

Authors: Hongliang Luo, Feifei Gao, Fan Liu, Shi Jin

Abstract: In this paper, we propose a novel scheme for sixdimensional (6D) radar sensing and tracking of dynamic target based on multiple input and multiple output (MIMO) array for monostatic integrated sensing and communications (ISAC) system. Unlike most existing ISAC studies believing that only the radial velocity of far-field dynamic target can be measured based on one single base station (BS), we find… ▽ More In this paper, we propose a novel scheme for sixdimensional (6D) radar sensing and tracking of dynamic target based on multiple input and multiple output (MIMO) array for monostatic integrated sensing and communications (ISAC) system. Unlike most existing ISAC studies believing that only the radial velocity of far-field dynamic target can be measured based on one single base station (BS), we find that the sensing echo channel of MIMO-ISAC system actually includes the distance, horizontal angle, pitch angle, radial velocity, horizontal angular velocity, and pitch angular velocity of the dynamic target. Thus we may fully rely on one single BS to estimate the dynamic target's 6D motion parameters from the sensing echo signals. Specifically, we first propose the long-term motion and short-term motion model of dynamic target, in which the short-term motion model serves the single-shot sensing of dynamic target, while the long-term motion model serves multiple-shots tracking of dynamic target. As a step further, we derive the sensing channel model corresponding to the short-term motion. Next, for singleshot sensing, we employ the array signal processing methods to estimate the dynamic target's horizontal angle, pitch angle, distance, and virtual velocity. By realizing that the virtual velocities observed by different antennas are different, we adopt plane fitting to estimate the radial velocity, horizontal angular velocity, and pitch angular velocity of dynamic target. Furthermore, we implement the multiple-shots tracking of dynamic target based on each single-shot sensing results and Kalman filtering. Simulation results demonstrate the effectiveness of the proposed 6D radar sensing and tracking scheme. △ Less

Submitted 27 December, 2023; originally announced December 2023.

Moving Target Sensing for ISAC Systems in Clutter Environment

Authors: Dongqi Luo, Huihui Wu, Hongliang Luo, Bo Lin, Feifei Gao

Abstract: In this paper, we consider the moving target sensing problem for integrated sensing and communication (ISAC) systems in clutter environment. Scatterers produce strong clutter, deteriorating the performance of ISAC systems in practice. Given that scatterers are typically stationary and the targets of interest are usually moving, we here focus on sensing the moving targets. Specifically, we adopt a… ▽ More In this paper, we consider the moving target sensing problem for integrated sensing and communication (ISAC) systems in clutter environment. Scatterers produce strong clutter, deteriorating the performance of ISAC systems in practice. Given that scatterers are typically stationary and the targets of interest are usually moving, we here focus on sensing the moving targets. Specifically, we adopt a scanning beam to search for moving target candidates. For the received signal in each scan, we employ high-pass filtering in the Doppler domain to suppress the clutter within the echo, thereby identifying candidate moving targets according to the power of filtered signal. Then, we adopt root-MUSIC-based algorithms to estimate the angle, range, and radial velocity of these candidate moving targets. Subsequently, we propose a target detection algorithm to reject false targets. Simulation results validate the effectiveness of these proposed methods. △ Less

Submitted 3 November, 2023; originally announced November 2023.

Integrated Sensing and Communications in Clutter Environment

Authors: Hongliang Luo, Yucong Wang, Dongqi Luo, Jianwei Zhao, Huihui Wu, Shaodan Ma, Feifei Gao

Abstract: In this paper, we propose a practical integrated sensing and communications (ISAC) framework to sense dynamic targets from clutter environment while ensuring users communications quality. To implement communications function and sensing function simultaneously, we design multiple communications beams that can communicate with the users as well as one sensing beam that can rotate and scan the entir… ▽ More In this paper, we propose a practical integrated sensing and communications (ISAC) framework to sense dynamic targets from clutter environment while ensuring users communications quality. To implement communications function and sensing function simultaneously, we design multiple communications beams that can communicate with the users as well as one sensing beam that can rotate and scan the entire space. To minimize the interference of sensing beam on existing communications systems, we divide the service area into sensing beam for sensing (S4S) sector and communications beam for sensing (C4S) sector, and provide beamforming design and power allocation optimization strategies for each type sector. Unlike most existing ISAC studies that ignore the interference of static environmental clutter on target sensing, we construct a mixed sensing channel model that includes both static environment and dynamic targets. When base station receives the echo signals, the mean phasor cancellation (MPC) method is employed to filter out the interference from static environmental clutter and to extract the effective dynamic target echoes. Then a complete and practical dynamic target sensing scheme is designed to detect the presence of dynamic targets and to estimate their angles, distances, and velocities. In particular, dynamic target detection and angle estimation are realized through angle-Doppler spectrum estimation (ADSE) and joint detection over multiple subcarriers (MSJD), while distance and velocity estimation are realized through the extended subspace algorithm. Simulation results demonstrate the effectiveness of the proposed scheme and its superiority over the existing methods that ignore environmental clutter. △ Less

Submitted 5 February, 2024; v1 submitted 2 November, 2023; originally announced November 2023.

Deep Learning Enables Large Depth-of-Field Images for Sub-Diffraction-Limit Scanning Superlens Microscopy

Authors: Hui Sun, Hao Luo, Feifei Wang, Qingjiu Chen, Meng Chen, Xiaoduo Wang, Haibo Yu, Guanglie Zhang, Lianqing Liu, Jianping Wang, Dapeng Wu, Wen Jung Li

Abstract: Scanning electron microscopy (SEM) is indispensable in diverse applications ranging from microelectronics to food processing because it provides large depth-of-field images with a resolution beyond the optical diffraction limit. However, the technology requires coating conductive films on insulator samples and a vacuum environment. We use deep learning to obtain the mapping relationship between op… ▽ More Scanning electron microscopy (SEM) is indispensable in diverse applications ranging from microelectronics to food processing because it provides large depth-of-field images with a resolution beyond the optical diffraction limit. However, the technology requires coating conductive films on insulator samples and a vacuum environment. We use deep learning to obtain the mapping relationship between optical super-resolution (OSR) images and SEM domain images, which enables the transformation of OSR images into SEM-like large depth-of-field images. Our custom-built scanning superlens microscopy (SSUM) system, which requires neither coating samples by conductive films nor a vacuum environment, is used to acquire the OSR images with features down to ~80 nm. The peak signal-to-noise ratio (PSNR) and structural similarity index measure values indicate that the deep learning method performs excellently in image-to-image translation, with a PSNR improvement of about 0.74 dB over the optical super-resolution images. The proposed method provides a high level of detail in the reconstructed results, indicating that it has broad applicability to chip-level defect detection, biological sample analysis, forensics, and various other fields. △ Less

Submitted 27 October, 2023; originally announced October 2023.

Comments: 13 pages,7 figures

Phase Synchrony Component Self-Organization in Brain Computer Interface

Authors: Xu Niu, Na Lu, Huan Luo, Ruofan Yan

Abstract: Phase synchrony information plays a crucial role in analyzing functional brain connectivity and identifying brain activities. A widely adopted feature extraction pipeline, composed of preprocessing, selection of EEG acquisition channels, and phase locking value (PLV) calculation, has achieved success in motor imagery classification (MI). However, this pipeline is manual and reliant on expert knowl… ▽ More Phase synchrony information plays a crucial role in analyzing functional brain connectivity and identifying brain activities. A widely adopted feature extraction pipeline, composed of preprocessing, selection of EEG acquisition channels, and phase locking value (PLV) calculation, has achieved success in motor imagery classification (MI). However, this pipeline is manual and reliant on expert knowledge, limiting its convenience and adaptability to different application scenarios. Moreover, most studies have employed mediocre data-independent spatial filters to suppress noise, impeding the exploration of more significant phase synchronization phenomena. To address the issues, we propose the concept of phase synchrony component self-organization, which enables the adaptive learning of data-dependent spatial filters for automating both the preprocessing and channel selection procedures. Based on this concept, the first deep learning end-to-end network is developed, which directly extracts phase synchrony-based features from raw EEG signals and perform classification. The network learns optimal filters during training, which are obtained when the network achieves peak classification results. Extensive experiments have demonstrated that our network outperforms state-of-the-art methods. Remarkably, through the learned optimal filters, significant phase synchronization phenomena can be observed. Specifically, by calculating the PLV between a pair of signals extracted from each sample using two of the learned spatial filters, we have obtained an average PLV exceeding 0.87 across all tongue MI samples. This high PLV indicates a groundbreaking discovery in the synchrony pattern of tongue MI. △ Less

Submitted 11 October, 2023; v1 submitted 21 September, 2023; originally announced October 2023.

Joint Audio and Speech Understanding

Authors: Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, James Glass

Abstract: Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perce… ▽ More Humans are surrounded by audio signals that include both speech and non-speech sounds. The recognition and understanding of speech and non-speech audio events, along with a profound comprehension of the relationship between them, constitute fundamental cognitive capabilities. For the first time, we build a machine learning model, called LTU-AS, that has a conceptually similar universal audio perception and advanced reasoning ability. Specifically, by integrating Whisper as a perception module and LLaMA as a reasoning module, LTU-AS can simultaneously recognize and jointly understand spoken text, speech paralinguistics, and non-speech audio events - almost everything perceivable from audio signals. △ Less

Submitted 10 December, 2023; v1 submitted 25 September, 2023; originally announced September 2023.

Comments: Accepted at ASRU 2023. Code, dataset, and pretrained models are at https://github.com/yuangongnd/ltu. Interactive demo at https://huggingface.co/spaces/yuangongfdu/ltu-2

Beam Squint Assisted User Localization in Near-Field Integrated Sensing and Communications Systems

Authors: Hongliang Luo, Feifei Gao, Wanmai Yuan, Shun Zhang

Abstract: Integrated sensing and communication (ISAC) has been regarded as a key technology for 6G wireless communications, in which large-scale multiple input and multiple output (MIMO) array with higher and wider frequency bands will be adopted. However, recent studies show that the beam squint phenomenon can not be ignored in wideband MIMO system, which generally deteriorates the communications performan… ▽ More Integrated sensing and communication (ISAC) has been regarded as a key technology for 6G wireless communications, in which large-scale multiple input and multiple output (MIMO) array with higher and wider frequency bands will be adopted. However, recent studies show that the beam squint phenomenon can not be ignored in wideband MIMO system, which generally deteriorates the communications performance. In this paper, we find that with the aid of true-time-delay lines (TTDs), the range and trajectory of the beam squint in near-field communications systems can be freely controlled, and hence it is possible to reversely utilize the beam squint for user localization. We derive the trajectory equation for near-field beam squint points and design a way to control such trajectory. With the proposed design, beamforming from different subcarriers would purposely point to different angles and different distances, such that users from different positions would receive the maximum power at different subcarriers. Hence, one can simply localize multiple users from the beam squint effect in frequency domain, and thus reduce the beam sweeping overhead as compared to the conventional time domain beam search based approach. Furthermore, we utilize the phase difference of the maximum power subcarriers received by the user at different frequencies in several times beam sweeping to obtain a more accurate distance estimation result, ultimately realizing high accuracy and low beam sweeping overhead user localization. Simulation results demonstrate the effectiveness of the proposed schemes. △ Less

Submitted 25 September, 2023; originally announced September 2023.

Comments: This paper has been accepted by IEEE Transactions on Wireless Communications (TWC) on 18 September 2023

Millimeter Wave V2V Beam Tracking using Radar: Algorithms and Real-World Demonstration

Authors: Hao Luo, Umut Demirhan, Ahmed Alkhateeb

Abstract: Utilizing radar sensing for assisting communication has attracted increasing interest thanks to its potential in dynamic environments. A particularly interesting problem for this approach appears in the vehicle-to-vehicle (V2V) millimeter wave and terahertz communication scenarios, where the narrow beams change with the movement of both vehicles. To address this problem, in this work, we develop a… ▽ More Utilizing radar sensing for assisting communication has attracted increasing interest thanks to its potential in dynamic environments. A particularly interesting problem for this approach appears in the vehicle-to-vehicle (V2V) millimeter wave and terahertz communication scenarios, where the narrow beams change with the movement of both vehicles. To address this problem, in this work, we develop a radar-aided beam-tracking framework, where a single initial beam and a set of radar measurements over a period of time are utilized to predict the future beams after this time duration. Within this framework, we develop two approaches with the combination of various degrees of radar signal processing and machine learning. To evaluate the feasibility of the solutions in a realistic scenario, we test their performance on a real-world V2V dataset. Our results indicated the importance of high angular resolution radar for this task and affirmed the potential of using radar for the V2V beam management problems. △ Less

Submitted 27 October, 2023; v1 submitted 3 August, 2023; originally announced August 2023.

Comments: 5 pages, 5 figures. To appear in EUSIPCO 2023. The dataset is available on the DeepSense 6G website http://deepsense6g.net/

YOLO: An Efficient Terahertz Band Integrated Sensing and Communications Scheme with Beam Squint

Authors: Hongliang Luo, Feifei Gao, Hai Lin, Shaodan Ma, H. Vincent Poor

Abstract: Using communications signals for dynamic target sensing is an important component of integrated sensing and communications (ISAC). In this paper, we propose to utilize the beam squint effect to realize fast non-cooperative dynamic target sensing in massive multiple input and multiple output (MIMO) Terahertz band communications systems. Specifically, we construct a wideband channel model of the ech… ▽ More Using communications signals for dynamic target sensing is an important component of integrated sensing and communications (ISAC). In this paper, we propose to utilize the beam squint effect to realize fast non-cooperative dynamic target sensing in massive multiple input and multiple output (MIMO) Terahertz band communications systems. Specifically, we construct a wideband channel model of the echo signals, and design a beamforming strategy that controls the range of beam squint by adjusting the values of phase shifters and true time delay lines. With this design, beams at different subcarriers can be aligned along different directions in a planned way. Then the received echo signals at different subcarriers will carry target information in different directions, based on which the targets' angles can be estimated through sophisticatedly designed algorithm. Moreover, we propose a supporting method based on extended array signal estimation, which utilizes the phase changes of different frequency subcarriers within different OFDM symbols to estimate the distance and velocity of dynamic targets. Interestingly, the proposed sensing scheme only needs to transmit and receive the signals once, which can be termed as You Only Listen Once (YOLO). Compared with the traditional ISAC method that requires time consuming beam sweeping, the proposed one greatly reduces the sensing overhead. Simulation results are provided to demonstrate the effectiveness of the proposed scheme. △ Less

Submitted 5 February, 2024; v1 submitted 19 May, 2023; originally announced May 2023.

Comments: This paper has been accepted by IEEE Transactions on Wireless Communications (TWC)

FunASR: A Fundamental End-to-End Speech Recognition Toolkit

Authors: Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, Shiliang Zhang

Abstract: This paper introduces FunASR, an open-source speech recognition toolkit designed to bridge the gap between academic research and industrial applications. FunASR offers models trained on large-scale industrial corpora and the ability to deploy them in applications. The toolkit's flagship model, Paraformer, is a non-autoregressive end-to-end speech recognition model that has been trained on a manual… ▽ More This paper introduces FunASR, an open-source speech recognition toolkit designed to bridge the gap between academic research and industrial applications. FunASR offers models trained on large-scale industrial corpora and the ability to deploy them in applications. The toolkit's flagship model, Paraformer, is a non-autoregressive end-to-end speech recognition model that has been trained on a manually annotated Mandarin speech recognition dataset that contains 60,000 hours of speech. To improve the performance of Paraformer, we have added timestamp prediction and hotword customization capabilities to the standard Paraformer backbone. In addition, to facilitate model deployment, we have open-sourced a voice activity detection model based on the Feedforward Sequential Memory Network (FSMN-VAD) and a text post-processing punctuation model based on the controllable time-delay Transformer (CT-Transformer), both of which were trained on industrial corpora. These functional modules provide a solid foundation for building high-precision long audio speech recognition services. Compared to other models trained on open datasets, Paraformer demonstrates superior performance. △ Less

Submitted 18 May, 2023; originally announced May 2023.

Comments: 5 pages, 3 figures, accepted by INTERSPEECH 2023

Listen, Think, and Understand

Authors: Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, James Glass

Abstract: The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general cat… ▽ More The ability of artificial intelligence (AI) systems to perceive and comprehend audio signals is crucial for many applications. Although significant progress has been made in this area since the development of AudioSet, most existing models are designed to map audio inputs to pre-defined, discrete sound label sets. In contrast, humans possess the ability to not only classify sounds into general categories, but also to listen to the finer details of the sounds, explain the reason for the predictions, think about what the sound infers, and understand the scene and what action needs to be taken, if any. Such capabilities beyond perception are not yet present in existing audio models. On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities. Therefore, we ask the question: can we build a model that has both audio perception and a reasoning ability? In this paper, we propose a new audio foundation model, called LTU (Listen, Think, and Understand). To train LTU, we created a new OpenAQA-5M dataset consisting of 1.9 million closed-ended and 3.7 million open-ended, diverse (audio, question, answer) tuples, and have used an autoregressive training framework with a perception-to-understanding curriculum. LTU demonstrates strong performance and generalization ability on conventional audio tasks such as classification and captioning. More importantly, it exhibits emerging audio reasoning and comprehension abilities that are absent in existing audio models. To the best of our knowledge, LTU is one of the first multimodal large language models that focus on general audio (rather than just speech) understanding. △ Less

Submitted 19 February, 2024; v1 submitted 18 May, 2023; originally announced May 2023.

Comments: Accepted at ICLR 2024. Code, dataset, and models are available at https://github.com/YuanGongND/ltu. The interactive demo is at https://huggingface.co/spaces/yuangongfdu/ltu

Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System

Authors: Xian Shi, Haoneng Luo, Zhifu Gao, Shiliang Zhang, Zhijie Yan

Abstract: Estimating confidence scores for recognition results is a classic task in ASR field and of vital importance for kinds of downstream tasks and training strategies. Previous end-to-end~(E2E) based confidence estimation models (CEM) predict score sequences of equal length with input transcriptions, leading to unreliable estimation when deletion and insertion errors occur. In this paper we proposed CI… ▽ More Estimating confidence scores for recognition results is a classic task in ASR field and of vital importance for kinds of downstream tasks and training strategies. Previous end-to-end~(E2E) based confidence estimation models (CEM) predict score sequences of equal length with input transcriptions, leading to unreliable estimation when deletion and insertion errors occur. In this paper we proposed CIF-Aligned confidence estimation model (CA-CEM) to achieve accurate and reliable confidence estimation based on novel non-autoregressive E2E ASR model - Paraformer. CA-CEM utilizes the modeling character of continuous integrate-and-fire (CIF) mechanism to generate token-synchronous acoustic embedding, which solves the estimation failure issue above. We measure the quality of estimation with AUC and RMSE in token level and ECE-U - a proposed metrics in utterance level. CA-CEM gains 24% and 19% relative reduction on ECE-U and also better AUC and RMSE on two test sets. Furthermore, we conduct analysis to explore the potential of CEM for different ASR related usage. △ Less

Submitted 24 May, 2023; v1 submitted 17 May, 2023; originally announced May 2023.

Comments: 5 pages, 4 figures, Interspeech2023

Inflation Reduction Act impacts on the economics of clean hydrogen and liquid fuels

Authors: Fangwei Cheng, Hongxi Luo, Jesse D. Jenkins, Eric D. Larson

Abstract: The Inflation Reduction Act (IRA) in the United States provides unprecedented incentives for deploying low-carbon hydrogen and liquid fuels, among other low greenhouse gas (GHG) emissions technologies. To better understand the prospective competitiveness of low-carbon or negative-carbon hydrogen and liquid fuels under the IRA in the early 2030s, we examine the impacts of IRA provisions on costs of… ▽ More The Inflation Reduction Act (IRA) in the United States provides unprecedented incentives for deploying low-carbon hydrogen and liquid fuels, among other low greenhouse gas (GHG) emissions technologies. To better understand the prospective competitiveness of low-carbon or negative-carbon hydrogen and liquid fuels under the IRA in the early 2030s, we examine the impacts of IRA provisions on costs of producing hydrogen and synthetic liquid fuel made from natural gas, electricity, short-cycle biomass (agricultural residues), and corn-ethanol. With IRA credits (45V or 45Q), but excluding incentives provided by other national or state policies, hydrogen produced by electrolysis using carbon-free electricity (green H2) and natural gas reforming with carbon capture and storage (CCS) (blue H2) are cost-competitive with the carbon-intensive benchmark gray H2 from steam methane reforming. Biomass-derived H2 with or without CCS is not cost-completive under current IRA provisions. However, if IRA allowed biomass gasification with CCS to claim a 45V credit for carbon-neutral H2 and a 45Q credit for negative biogenic-CO2 emissions, this pathway would be less costly than gray H2. The IRA credit for clean fuels (45Z), currently stipulated to end in 2027, would need to be extended, or similar policy support provided by other national or state policies, for clean synthetic liquid fuel to be cost-competitive with petroleum-derived liquid fuels. Levelized IRA subsidies per unit of CO2 mitigated for all hydrogen and synthetic liquid fuel production pathways, except electricity-derived synthetic liquid fuel, range from 65 to 384 $/t CO2, which is within or below the range in U.S. federal government estimates of the Social Cost of Carbon (SCC) in the 2030 to 2040 timeframe. △ Less

Submitted 14 August, 2023; v1 submitted 1 May, 2023; originally announced May 2023.

ESCM: An Efficient and Secure Communication Mechanism for UAV Networks

Authors: Haoxiang Luo, Yifan Wu, Gang Sun, Hongfang Yu, Mohsen Guizani

Abstract: UAV (unmanned aerial vehicle) is rapidly gaining traction in various human activities and has become an integral component of the satellite-air-ground-sea (SAGS) integrated network. As high-speed moving objects, UAVs not only have extremely strict requirements for communication delay, but also cannot be maliciously controlled as a weapon by the attacker. Therefore, an efficient and secure communic… ▽ More UAV (unmanned aerial vehicle) is rapidly gaining traction in various human activities and has become an integral component of the satellite-air-ground-sea (SAGS) integrated network. As high-speed moving objects, UAVs not only have extremely strict requirements for communication delay, but also cannot be maliciously controlled as a weapon by the attacker. Therefore, an efficient and secure communication method designed for UAV networks is necessary. We propose a communication mechanism ESCM. For high efficiency, ESCM provides a routing protocol based on the artificial bee colony (ABC) algorithm to accelerate communications between UAVs. Meanwhile, we use blockchain to guarantee the security of UAV networks. However, blockchain has unstable links in high-mobility networks resulting in low consensus efficiency and high communication overhead. Consequently, ESCM introduces digital twin (DT), which transforms the UAV network into a static network by mapping UAVs from the physical world into Cyberspace. This virtual UAV network is called CyberUAV. Then, in CyberUAV, we design a blockchain consensus based on network coding, named Proof of Network Coding (PoNC). Analysis and simulation show that the above modules in ESCM have advantages over existing schemes. Through ablation studies, we demonstrate that these modules are indispensable for efficient and secure communication of UAV networks. △ Less

Submitted 16 June, 2023; v1 submitted 25 April, 2023; originally announced April 2023.

Performance Analysis and Comparison of Non-ideal Wireless PBFT and RAFT Consensus Networks in 6G Communications

Authors: Haoxiang Luo, Xiangyue Yang, Hongfang Yu, Gang Sun, Bo Lei, Mohsen Guizani

Abstract: Due to advantages in security and privacy, blockchain is considered a key enabling technology to support 6G communications. Practical Byzantine Fault Tolerance (PBFT) and RAFT are seen as the most applicable consensus mechanisms (CMs) in blockchain-enabled wireless networks. However, previous studies on PBFT and RAFT rarely consider the channel performance of the physical layer, such as path loss… ▽ More Due to advantages in security and privacy, blockchain is considered a key enabling technology to support 6G communications. Practical Byzantine Fault Tolerance (PBFT) and RAFT are seen as the most applicable consensus mechanisms (CMs) in blockchain-enabled wireless networks. However, previous studies on PBFT and RAFT rarely consider the channel performance of the physical layer, such as path loss and channel fading, resulting in research results that are far from real networks. Additionally, 6G communications will widely deploy high-frequency signals such as terahertz (THz) and millimeter wave (mmWave), while performances of PBFT and RAFT are still unknown when these signals are transmitted in wireless PBFT or RAFT networks. Therefore, it is urgent to study the performance of non-ideal wireless PBFT and RAFT networks with THz and mmWave signals, to better make PBFT and RAFT play a role in the 6G era. In this paper, we study and compare the performance of THz and mmWave signals in non-ideal wireless PBFT and RAFT networks, considering Rayleigh Fading (RF) and close-in Free Space (FS) reference distance path loss. Performance is evaluated by five metrics: consensus success rate, latency, throughput, reliability gain, and energy consumption. Meanwhile, we find and derive that there is a maximum distance between two nodes that can make CMs inevitably successful, and it is named the active distance of CMs. The research results analyze the performance of non-ideal wireless PBFT and RAFT networks, and provide important references for the future transmission of THz and mmWave signals in PBFT and RAFT networks. △ Less

Submitted 2 August, 2023; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: arXiv admin note: substantial text overlap with arXiv:2303.15759

Performance Analysis of Non-ideal Wireless PBFT Networks with mmWave and Terahertz Signals

Authors: Haoxiang Luo, Xiangyue Yang, Hongfang Yu, Gang Sun, Shizhong Xu, Long Luo

Abstract: Due to advantages in security and privacy, blockchain is considered a key enabling technology to support 6G communications. Practical Byzantine Fault Tolerance (PBFT) is seen as the most applicable consensus mechanism in blockchain-enabled wireless networks. However, previous studies on PBFT do not consider the channel performance of the physical layer, such as path loss and channel fading, result… ▽ More Due to advantages in security and privacy, blockchain is considered a key enabling technology to support 6G communications. Practical Byzantine Fault Tolerance (PBFT) is seen as the most applicable consensus mechanism in blockchain-enabled wireless networks. However, previous studies on PBFT do not consider the channel performance of the physical layer, such as path loss and channel fading, resulting in research results that are far from real networks. Additionally, 6G communications will widely deploy high frequency signals such as millimeter wave (mmWave) and terahertz (THz), while the performance of PBFT is still unknown when these signals are transmitted in wireless PBFT networks. Therefore, it is urgent to study the performance of non-ideal wireless PBFT networks with mmWave and THz siganls, so as to better make PBFT play a role in 6G era. In this paper, we study and compare the performance of mmWave and THz signals in non-ideal wireless PBFT networks, considering Rayleigh Fading (RF) and close-in Free Space (FS) reference distance path loss. Performance is evaluated by consensus success rate and delay. Meanwhile, we find and derive that there is a maximum distance between two nodes that can make PBFT consensus inevitably successful, and it is named active distance of PBFT in this paper. The research results not only analyze the performance of non-ideal wireless PBFT networks, but also provide an important reference for the future transmission of mmWave and THz signals in PBFT networks. △ Less

Submitted 28 March, 2023; originally announced March 2023.

Comments: IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom) 2023

RIS-Aided Integrated Sensing and Communication: Joint Beamforming and Reflection Design

Authors: Honghao Luo, Rang Liu, Ming Li, Qian Liu

Abstract: Integrated sensing and communication (ISAC) has been envisioned as a promising technique to alleviate the spectrum congestion problem. Inspired by the applications of reconfigurable intelligent surface (RIS) in dynamically manipulating wireless propagation environment, in this paper, we investigate to deploy a RIS in an ISAC system to pursue performance improvement. Particularly, we consider a RIS… ▽ More Integrated sensing and communication (ISAC) has been envisioned as a promising technique to alleviate the spectrum congestion problem. Inspired by the applications of reconfigurable intelligent surface (RIS) in dynamically manipulating wireless propagation environment, in this paper, we investigate to deploy a RIS in an ISAC system to pursue performance improvement. Particularly, we consider a RIS-assisted ISAC system where a multi-antenna base station (BS) performs multi-target detection and multi-user communication with the assistance of a RIS. Our goal is maximizing the weighted summation of target detection signal-to-noise ratios (SNRs) by jointly optimizing the transmit beamforming and the RIS reflection coefficients, while satisfying the communication quality-of-service (QoS) requirement, the total transmit power budget, and the restriction of RIS phase-shift. An efficient alternating optimization algorithm combining the majorization-minimization (MM), penalty-based, and manifold optimization methods is developed to solve the resulting complicated non-convex optimization problem. Simulation results illustrate the advantages of deploying RIS in ISAC systems and the effectiveness of our proposed algorithm. △ Less

Submitted 22 February, 2023; originally announced February 2023.

Comments: Accepted by IEEE TVT

Interpretable Spectrum Transformation Attacks to Speaker Recognition

Authors: Jiadi Yao, Hong Luo, Xiao-Lei Zhang

Abstract: The success of adversarial attacks to speaker recognition is mainly in white-box scenarios. When applying the adversarial voices that are generated by attacking white-box surrogate models to black-box victim models, i.e. \textit{transfer-based} black-box attacks, the transferability of the adversarial voices is not only far from satisfactory, but also lacks interpretable basis. To address these is… ▽ More The success of adversarial attacks to speaker recognition is mainly in white-box scenarios. When applying the adversarial voices that are generated by attacking white-box surrogate models to black-box victim models, i.e. \textit{transfer-based} black-box attacks, the transferability of the adversarial voices is not only far from satisfactory, but also lacks interpretable basis. To address these issues, in this paper, we propose a general framework, named spectral transformation attack based on modified discrete cosine transform (STA-MDCT), to improve the transferability of the adversarial voices to a black-box victim model. Specifically, we first apply MDCT to the input voice. Then, we slightly modify the energy of different frequency bands for capturing the salient regions of the adversarial noise in the time-frequency domain that are critical to a successful attack. Unlike existing approaches that operate voices in the time domain, the proposed framework operates voices in the time-frequency domain, which improves the interpretability, transferability, and imperceptibility of the attack. Moreover, it can be implemented with any gradient-based attackers. To utilize the advantage of model ensembling, we not only implement STA-MDCT with a single white-box surrogate model, but also with an ensemble of surrogate models. Finally, we visualize the saliency maps of adversarial voices by the class activation maps (CAM), which offers an interpretable basis to transfer-based attacks in speaker recognition for the first time. Extensive comparison results with five representative attackers show that the CAM visualization clearly explains the effectiveness of STA-MDCT, and the weaknesses of the comparison methods; the proposed method outperforms the comparison methods by a large margin. △ Less

Submitted 21 February, 2023; originally announced February 2023.

Incipient Fault Detection in Power Distribution System: A Time-Frequency Embedded Deep Learning Based Approach

Authors: Qiyue Li, Huan Luo, Hong Cheng, Yuxing Deng, Wei Sun, Weitao Li, Zhi Liu

Abstract: Incipient fault detection in power distribution systems is crucial to improve the reliability of the grid. However, the non-stationary nature and the inadequacy of the training dataset due to the self-recovery of the incipient fault signal, make the incipient fault detection in power distribution systems a great challenge. In this paper, we focus on incipient fault detection in power distribution… ▽ More Incipient fault detection in power distribution systems is crucial to improve the reliability of the grid. However, the non-stationary nature and the inadequacy of the training dataset due to the self-recovery of the incipient fault signal, make the incipient fault detection in power distribution systems a great challenge. In this paper, we focus on incipient fault detection in power distribution systems and address the above challenges. In particular, we propose an ADaptive Time-Frequency Memory(AD-TFM) cell by embedding wavelet transform into the Long Short-Term Memory (LSTM), to extract features in time and frequency domain from the non-stationary incipient fault signals.We make scale parameters and translation parameters of wavelet transform learnable to adapt to the dynamic input signals. Based on the stacked AD-TFM cells, we design a recurrent neural network with ATtention mechanism, named AD-TFM-AT model, to detect incipient fault with multi-resolution and multi-dimension analysis. In addition, we propose two data augmentation methods, namely phase switching and temporal sliding, to effectively enlarge the training datasets. Experimental results on two open datasets show that our proposed AD-TFM-AT model and data augmentation methods achieve state-of-the-art (SOTA) performance of incipient fault detection in power distribution system. We also disclose one used dataset logged at State Grid Corporation of China to facilitate future research. △ Less

Submitted 18 February, 2023; originally announced February 2023.

Comments: 15 pages

Reinforcement learning for traffic signal control in hybrid action space

Authors: Haoqing Luo, sheng jin

Abstract: The prevailing reinforcement-learning-based traffic signal control methods are typically staging-optimizable or duration-optimizable, depending on the action spaces. In this paper, we propose a novel control architecture, TBO, which is based on hybrid proximal policy optimization. To the best of our knowledge, TBO is the first RL-based algorithm to implement synchronous optimization of the staging… ▽ More The prevailing reinforcement-learning-based traffic signal control methods are typically staging-optimizable or duration-optimizable, depending on the action spaces. In this paper, we propose a novel control architecture, TBO, which is based on hybrid proximal policy optimization. To the best of our knowledge, TBO is the first RL-based algorithm to implement synchronous optimization of the staging and duration. Compared to discrete and continuous action spaces, hybrid action space is a merged search space, in which TBO better implements the trade-off between frequent switching and unsaturated release. Experiments are given to demonstrate that TBO reduces the queue length and delay by 13.78% and 14.08% on average, respectively, compared to the existing baselines. Furthermore, we calculate the Gini coefficients of the right-of-way to indicate TBO does not harm fairness while improving efficiency. △ Less

Submitted 25 November, 2022; v1 submitted 23 November, 2022; originally announced November 2022.

Comments: There are serious problems with the innovation of the paper

Reconfigurable Intelligent Surface Aided Wireless Sensing for Scene Depth Estimation

Authors: Abdelrahman Taha, Hao Luo, Ahmed Alkhateeb

Abstract: Current scene depth estimation approaches mainly rely on optical sensing, which carries privacy concerns and suffers from estimation ambiguity for distant, shiny, and transparent surfaces/objects. Reconfigurable intelligent surfaces (RISs) provide a path for employing a massive number of antennas using low-cost and energy-efficient architectures. This has the potential for realizing RIS-aided wire… ▽ More Current scene depth estimation approaches mainly rely on optical sensing, which carries privacy concerns and suffers from estimation ambiguity for distant, shiny, and transparent surfaces/objects. Reconfigurable intelligent surfaces (RISs) provide a path for employing a massive number of antennas using low-cost and energy-efficient architectures. This has the potential for realizing RIS-aided wireless sensing with high spatial resolution. In this paper, we propose to employ RIS-aided wireless sensing systems for scene depth estimation. We develop a comprehensive framework for building accurate depth maps using RIS-aided mmWave sensing systems. In this framework, we propose a new RIS interaction codebook capable of creating a sensing grid of reflected beams that meets the desirable characteristics of efficient scene depth map construction. Using the designed codebook, the received signals are processed to build high-resolution depth maps. Simulation results compare the proposed solution against RGB-based approaches and highlight the promise of adopting RIS-aided mmWave sensing in scene depth perception. △ Less

Submitted 15 November, 2022; originally announced November 2022.

Comments: Submitted to IEEE

3D Matting: A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography

Authors: Lin Wang, Xiufen Ye, Donghao Zhang, Wanji He, Lie Ju, Yi Luo, Huan Luo, Xin Wang, Wei Feng, Kaimin Song, Xin Zhao, Zongyuan Ge

Abstract: Usually, lesions are not isolated but are associated with the surrounding tissues. For example, the growth of a tumour can depend on or infiltrate into the surrounding tissues. Due to the pathological nature of the lesions, it is challenging to distinguish their boundaries in medical imaging. However, these uncertain regions may contain diagnostic information. Therefore, the simple binarization of… ▽ More Usually, lesions are not isolated but are associated with the surrounding tissues. For example, the growth of a tumour can depend on or infiltrate into the surrounding tissues. Due to the pathological nature of the lesions, it is challenging to distinguish their boundaries in medical imaging. However, these uncertain regions may contain diagnostic information. Therefore, the simple binarization of lesions by traditional binary segmentation can result in the loss of diagnostic information. In this work, we introduce the image matting into the 3D scenes and use the alpha matte, i.e., a soft mask, to describe lesions in a 3D medical image. The traditional soft mask acted as a training trick to compensate for the easily mislabelled or under-labelled ambiguous regions. In contrast, 3D matting uses soft segmentation to characterize the uncertain regions more finely, which means that it retains more structural information for subsequent diagnosis and treatment. The current study of image matting methods in 3D is limited. To address this issue, we conduct a comprehensive study of 3D matting, including both traditional and deep-learning-based methods. We adapt four state-of-the-art 2D image matting algorithms to 3D scenes and further customize the methods for CT images to calibrate the alpha matte with the radiodensity. Moreover, we propose the first end-to-end deep 3D matting network and implement a solid 3D medical image matting benchmark. Its efficient counterparts are also proposed to achieve a good performance-computation balance. Furthermore, there is no high-quality annotated dataset related to 3D matting, slowing down the development of data-driven deep-learning-based methods. To address this issue, we construct the first 3D medical matting dataset. The validity of the dataset was verified through clinicians' assessments and downstream experiments. △ Less

Submitted 10 October, 2022; originally announced October 2022.

Comments: Accepted by Computers in Biology and Medicine. arXiv admin note: substantial text overlap with arXiv:2209.07843

Joint Beamforming Design for RIS-Assisted Integrated Sensing and Communication Systems

Authors: Honghao Luo, Rang Liu, Ming Li, Yang Liu, Qian Liu

Abstract: Integrated sensing and communication (ISAC) has been envisioned as a promising technology to tackle the spectrum congestion problem for future networks. In this correspondence, we investigate to deploy a reconfigurable intelligent surface (RIS) in an ISAC system for achieving better performance. In particular, a multi-antenna base station (BS) simultaneously serves multiple single-antenna users wi… ▽ More Integrated sensing and communication (ISAC) has been envisioned as a promising technology to tackle the spectrum congestion problem for future networks. In this correspondence, we investigate to deploy a reconfigurable intelligent surface (RIS) in an ISAC system for achieving better performance. In particular, a multi-antenna base station (BS) simultaneously serves multiple single-antenna users with the assistance of a RIS and detects potential targets. The active beamforming of the BS and the passive beamforming of the RIS are jointly optimized to maximize the achievable sum-rate of the communication users while satisfying the constraint of beampattern similarity for radar sensing, the restriction of the RIS, and the transmit power budget. An efficient alternating algorithm based on the fractional programming (FP), majorization-minimization (MM), and manifold optimization methods is developed to convert the resulting non-convex optimization problem into two solvable sub-problems and iteratively solve them. Simulation studies illustrate the advancement of deploying RIS in ISAC systems and the effectiveness of the proposed algorithm. △ Less

Submitted 3 August, 2022; originally announced August 2022.

Comments: Accepted by IEEE TVT

Weakly-supervised High-fidelity Ultrasound Video Synthesis with Feature Decoupling

Authors: Jiamin Liang, Xin Yang, Yuhao Huang, Kai Liu, Xinrui Zhou, Xindi Hu, Zehui Lin, Huanjia Luo, Yuanji Zhang, Yi Xiong, Dong Ni

Abstract: Ultrasound (US) is widely used for its advantages of real-time imaging, radiation-free and portability. In clinical practice, analysis and diagnosis often rely on US sequences rather than a single image to obtain dynamic anatomical information. This is challenging for novices to learn because practicing with adequate videos from patients is clinically unpractical. In this paper, we propose a novel… ▽ More Ultrasound (US) is widely used for its advantages of real-time imaging, radiation-free and portability. In clinical practice, analysis and diagnosis often rely on US sequences rather than a single image to obtain dynamic anatomical information. This is challenging for novices to learn because practicing with adequate videos from patients is clinically unpractical. In this paper, we propose a novel framework to synthesize high-fidelity US videos. Specifically, the synthesis videos are generated by animating source content images based on the motion of given driving videos. Our highlights are three-fold. First, leveraging the advantages of self- and fully-supervised learning, our proposed system is trained in weakly-supervised manner for keypoint detection. These keypoints then provide vital information for handling complex high dynamic motions in US videos. Second, we decouple content and texture learning using the dual decoders to effectively reduce the model learning difficulty. Last, we adopt the adversarial training strategy with GAN losses for further improving the sharpness of the generated videos, narrowing the gap between real and synthesis videos. We validate our method on a large in-house pelvic dataset with high dynamic motion. Extensive evaluation metrics and user study prove the effectiveness of our proposed method. △ Less

Submitted 1 July, 2022; originally announced July 2022.

Comments: Accepted by MICCAI 2022

Integrated Sensing and Communication with Reconfigurable Intelligent Surfaces: Opportunities, Applications, and Future Directions

Authors: Rang Liu, Ming Li, Honghao Luo, Qian Liu, A. Lee Swindlehurst

Abstract: Integrated sensing and communication (ISAC) is emerging as a key enabler to address the growing spectrum congestion problem and satisfy increasing demands for ubiquitous sensing and communication. By sharing various resources and information, ISAC achieves much higher spectral, energy, hardware, and economic efficiencies. Concurrently, reconfigurable intelligent surface (RIS) technology has been d… ▽ More Integrated sensing and communication (ISAC) is emerging as a key enabler to address the growing spectrum congestion problem and satisfy increasing demands for ubiquitous sensing and communication. By sharing various resources and information, ISAC achieves much higher spectral, energy, hardware, and economic efficiencies. Concurrently, reconfigurable intelligent surface (RIS) technology has been deemed as a promising approach due to its capability of intelligently manipulating the wireless propagation environment in an energy and hardware efficient manner. In this article, we analyze the potential of deploying RIS to improve communication and sensing performance in ISAC systems. We first describe the fundamentals of RIS and its applications in traditional communication and sensing systems, then introduce the principles of ISAC and overview existing explorations on RIS-assisted ISAC, followed by one case study to verify the advantages of deploying RIS in ISAC systems. Finally, open challenges and research directions are discussed to stimulate this line of research and pave the way for practical applications. △ Less

Submitted 16 June, 2022; originally announced June 2022.

Comments: submitted to IEEE journal

Beam Squint Assisted User Localization in Near-Field Communications Systems

Authors: Hongliang Luo, Feifei Gao

Abstract: The beam squint phenomenon in massive multi-input and multi-output wideband communications has been widely concerned recently, which generally deteriorates the beamforming performance. In this paper, we find that with the aid of the time-delay lines (TDs), the range and trajectory of the beam squint of a near-field communications system can be freely controlled, and hence it is possible to reverse… ▽ More The beam squint phenomenon in massive multi-input and multi-output wideband communications has been widely concerned recently, which generally deteriorates the beamforming performance. In this paper, we find that with the aid of the time-delay lines (TDs), the range and trajectory of the beam squint of a near-field communications system can be freely controlled, and hence it is possible to reversely utilize the beam squint for user localization. We derive the trajectory equation for near-field beam squint points and design a way to control the trajectory of these beam squint points. With the proposed design, beamforming from different subcarriers would purposely point to different angles and different distances such that users from different positions would receive the maximum power at different subcarriers. Hence, one can simply find the different users' position from the beam squint effect. Simulation results demonstrate the effectiveness of the proposed scheme. △ Less

Submitted 23 May, 2022; originally announced May 2022.