Showing 1–50 of 269 results for author: Zheng, Y

SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge

Authors: Yuxiang Mei, Yuang Zheng, Dongxing Xu, Yanhua Long

Abstract: This paper describes SHNU multilingual conversational speech recognition system (SHNU-mASR, team name-"maybe"), submitted to Track 1 of the INTERSPEECH 2025 MLC-SLM Challenge. Our system integrates a parallel-speech-encoder architecture with a large language model (LLM) to form a unified multilingual ASR framework. The parallel-speech-encoder consists of two pre-trained encoders, the Whisper-large… ▽ More This paper describes SHNU multilingual conversational speech recognition system (SHNU-mASR, team name-"maybe"), submitted to Track 1 of the INTERSPEECH 2025 MLC-SLM Challenge. Our system integrates a parallel-speech-encoder architecture with a large language model (LLM) to form a unified multilingual ASR framework. The parallel-speech-encoder consists of two pre-trained encoders, the Whisper-large-v3 encoder and mHuBERT-147 encoder. Their output embeddings are concatenated and fed into the LLM, enabling the model to leverage complementary acoustic and linguistic knowledge and achieve competitive performance. Moreover, we adopt a tri-stage training strategy to jointly update the low-rank adaptation modules and projector parameters of both the speech encoders and the LLM. In addition, we incorporate an additional language-aware prompt at the LLM input to enhance language-specific text generation. The SHNU-mASR system achieves an overall character/word error rate (CER/WER) of 11.76% on the blind evaluation set of the challenge, outperforming the official MLC-SLM baseline by 8.41 absolute CER/WER, without increasing the baseline training data. △ Less

Submitted 8 July, 2025; v1 submitted 4 July, 2025; originally announced July 2025.

Comments: Accepted by Interspeech 2025 MLC-SLM workshop

Hybrid Constellation Modulation for Symbol-Level Precoding in RIS-Enhanced MU-MISO Systems

Authors: Yupeng Zheng, Yi Ma, Rahim Tafazolli

Abstract: The application of symbol-level precoding (SLP) in reconfigurable intelligent surfaces (RIS) enhanced multi-user multiple-input single-output (MU-MISO) systems faces two main challenges. First, the state-of-the-art joint reflecting and SLP optimization approach requires exhaustive enumeration of all possible transmit symbol combinations, resulting in scalability issues as the modulation order and… ▽ More The application of symbol-level precoding (SLP) in reconfigurable intelligent surfaces (RIS) enhanced multi-user multiple-input single-output (MU-MISO) systems faces two main challenges. First, the state-of-the-art joint reflecting and SLP optimization approach requires exhaustive enumeration of all possible transmit symbol combinations, resulting in scalability issues as the modulation order and number of users increase. Second, conventional quadrature amplitude modulation (QAM) exhibits strict constructive interference (CI) regions, limiting its effectiveness for CI exploitation in SLP. To address these challenges, this paper proposes a novel modulation scheme, termed hybrid-constellation modulation (HCM), which has a structure of superposed QAM and ASK sub-constellations (SCs). HCM extends the CI regions compared to QAM. Additionally, a two-stage reflecting and SLP optimization method is developed to support HCM. The proposed methods are designed for practical RIS with discrete phase shifts and has good scalability. Simulation results show that HCM achieves up to 1.5 dB and 1 dB SER gains over QAM with modulation order 16 and 64, respectively. △ Less

Submitted 27 June, 2025; originally announced June 2025.

Comments: This work has been accepted by IEEE SPAWC 2025

Combining Self-attention and Dilation Convolutional for Semantic Segmentation of Coal Maceral Groups

Authors: Zhenghao Xi, Zhengnan Lv, Yang Zheng, Xiang Liu, Zhuang Yu, Junran Chen, Jing Hu, Yaqi Liu

Abstract: The segmentation of coal maceral groups can be described as a semantic segmentation process of coal maceral group images, which is of great significance for studying the chemical properties of coal. Generally, existing semantic segmentation models of coal maceral groups use the method of stacking parameters to achieve higher accuracy. It leads to increased computational requirements and impacts mo… ▽ More The segmentation of coal maceral groups can be described as a semantic segmentation process of coal maceral group images, which is of great significance for studying the chemical properties of coal. Generally, existing semantic segmentation models of coal maceral groups use the method of stacking parameters to achieve higher accuracy. It leads to increased computational requirements and impacts model training efficiency. At the same time, due to the professionalism and diversity of coal maceral group images sampling, obtaining the number of samples for model training requires a long time and professional personnel operation. To address these issues, We have innovatively developed an IoT-based DA-VIT parallel network model. By utilizing this model, we can continuously broaden the dataset through IoT and achieving sustained improvement in the accuracy of coal maceral groups segmentation. Besides, we decouple the parallel network from the backbone network to ensure the normal using of the backbone network during model data updates. Secondly, DCSA mechanism of DA-VIT is introduced to enhance the local feature information of coal microscopic images. This DCSA can decompose the large kernels of convolutional attention into multiple scales and reduce 81.18% of parameters.Finally, we performed the contrast experiment and ablation experiment between DA-VIT and state-of-the-art methods at lots of evaluation metrics. Experimental results show that DA-VIT-Base achieves 92.14% pixel accuracy and 63.18% mIoU. Params and FLOPs of DA-VIT-Tiny are 4.95M and 8.99G, respectively. All of the evaluation metrics of the proposed DA-VIT are better than other state-of-the-art methods. △ Less

Submitted 15 June, 2025; originally announced June 2025.

Rethinking Brain Tumor Segmentation from the Frequency Domain Perspective

Authors: Minye Shao, Zeyu Wang, Haoran Duan, Yawen Huang, Bing Zhai, Shizheng Wang, Yang Long, Yefeng Zheng

Abstract: Precise segmentation of brain tumors, particularly contrast-enhancing regions visible in post-contrast MRI (areas highlighted by contrast agent injection), is crucial for accurate clinical diagnosis and treatment planning but remains challenging. However, current methods exhibit notable performance degradation in segmenting these enhancing brain tumor areas, largely due to insufficient considerati… ▽ More Precise segmentation of brain tumors, particularly contrast-enhancing regions visible in post-contrast MRI (areas highlighted by contrast agent injection), is crucial for accurate clinical diagnosis and treatment planning but remains challenging. However, current methods exhibit notable performance degradation in segmenting these enhancing brain tumor areas, largely due to insufficient consideration of MRI-specific tumor features such as complex textures and directional variations. To address this, we propose the Harmonized Frequency Fusion Network (HFF-Net), which rethinks brain tumor segmentation from a frequency-domain perspective. To comprehensively characterize tumor regions, we develop a Frequency Domain Decomposition (FDD) module that separates MRI images into low-frequency components, capturing smooth tumor contours and high-frequency components, highlighting detailed textures and directional edges. To further enhance sensitivity to tumor boundaries, we introduce an Adaptive Laplacian Convolution (ALC) module that adaptively emphasizes critical high-frequency details using dynamically updated convolution kernels. To effectively fuse tumor features across multiple scales, we design a Frequency Domain Cross-Attention (FDCA) integrating semantic, positional, and slice-specific information. We further validate and interpret frequency-domain improvements through visualization, theoretical reasoning, and experimental analyses. Extensive experiments on four public datasets demonstrate that HFF-Net achieves an average relative improvement of 4.48\% (ranging from 2.39\% to 7.72\%) in the mean Dice scores across the three major subregions, and an average relative improvement of 7.33% (ranging from 5.96% to 8.64%) in the segmentation of contrast-enhancing tumor regions, while maintaining favorable computational efficiency and clinical applicability. Code: https://github.com/VinyehShaw/HFF. △ Less

Submitted 11 June, 2025; originally announced June 2025.

Comments: Accepted by IEEE Transactions on Medical Imaging

Dynamic real-time multi-UAV cooperative mission planning method under multiple constraints

Authors: Chenglou Liu, Yufeng Lu, Fangfang Xie, Tingwei Ji, Yao Zheng

Abstract: As UAV popularity soars, so does the mission planning associated with it. The classical approaches suffer from the triple problems of decoupled of task assignment and path planning, poor real-time performance and limited adaptability. Aiming at these challenges, this paper proposes a dynamic real-time multi-UAV collaborative mission planning algorithm based on Dubins paths under a distributed form… ▽ More As UAV popularity soars, so does the mission planning associated with it. The classical approaches suffer from the triple problems of decoupled of task assignment and path planning, poor real-time performance and limited adaptability. Aiming at these challenges, this paper proposes a dynamic real-time multi-UAV collaborative mission planning algorithm based on Dubins paths under a distributed formation structure. Dubins path with multiple advantages bridges the gap between task assignment and path planning, leading to a coupled solution for mission planning. Then, a series of acceleration techniques, task clustering preprocessing, highly efficient distance cost functions, low-complexity and less iterative task allocation strategies, are employed to guarantee the real-time performance of the algorithms. To cope with different emergencies and their simultaneous extremes, real-time planning of emerging tasks and mission replanning due to the reduction of available UAVs are appropriately handled. Finally, the developed algorithm is comprehensively exemplified and studied through simulations, highlighting that the proposed method only sacrifices 9.57% of the path length, while achieving a speed improvement of 4-5 orders of magnitude over the simulated annealing method, with a single mission planning of about 0.0003s. △ Less

Submitted 2 June, 2025; originally announced June 2025.

DarkDiff: Advancing Low-Light Raw Enhancement by Retasking Diffusion Models for Camera ISP

Authors: Amber Yijia Zheng, Yu Zhang, Jun Hu, Raymond A. Yeh, Chen Chen

Abstract: High-quality photography in extreme low-light conditions is challenging but impactful for digital cameras. With advanced computing hardware, traditional camera image signal processor (ISP) algorithms are gradually being replaced by efficient deep networks that enhance noisy raw images more intelligently. However, existing regression-based models often minimize pixel errors and result in oversmooth… ▽ More High-quality photography in extreme low-light conditions is challenging but impactful for digital cameras. With advanced computing hardware, traditional camera image signal processor (ISP) algorithms are gradually being replaced by efficient deep networks that enhance noisy raw images more intelligently. However, existing regression-based models often minimize pixel errors and result in oversmoothing of low-light photos or deep shadows. Recent work has attempted to address this limitation by training a diffusion model from scratch, yet those models still struggle to recover sharp image details and accurate colors. We introduce a novel framework to enhance low-light raw images by retasking pre-trained generative diffusion models with the camera ISP. Extensive experiments demonstrate that our method outperforms the state-of-the-art in perceptual quality across three challenging low-light raw image benchmarks. △ Less

Submitted 29 May, 2025; originally announced May 2025.

Boosting Adversarial Transferability via High-Frequency Augmentation and Hierarchical-Gradient Fusion

Authors: Yayin Zheng, Chen Wan, Zihong Guo, Hailing Kuang, Xiaohai Lu

Abstract: Adversarial attacks have become a significant challenge in the security of machine learning models, particularly in the context of black-box defense strategies. Existing methods for enhancing adversarial transferability primarily focus on the spatial domain. This paper presents Frequency-Space Attack (FSA), a new adversarial attack framework that effectively integrates frequency-domain and spatial… ▽ More Adversarial attacks have become a significant challenge in the security of machine learning models, particularly in the context of black-box defense strategies. Existing methods for enhancing adversarial transferability primarily focus on the spatial domain. This paper presents Frequency-Space Attack (FSA), a new adversarial attack framework that effectively integrates frequency-domain and spatial-domain transformations. FSA combines two key techniques: (1) High-Frequency Augmentation, which applies Fourier transform with frequency-selective amplification to diversify inputs and emphasize the critical role of high-frequency components in adversarial attacks, and (2) Hierarchical-Gradient Fusion, which merges multi-scale gradient decomposition and fusion to capture both global structures and fine-grained details, resulting in smoother perturbations. Our experiment demonstrates that FSA consistently outperforms state-of-the-art methods across various black-box models. Notably, our proposed FSA achieves an average attack success rate increase of 23.6% compared with BSR (CVPR 2024) on eight black-box defense models. △ Less

Submitted 27 May, 2025; originally announced May 2025.

SpecWav-Attack: Leveraging Spectrogram Resizing and Wav2Vec 2.0 for Attacking Anonymized Speech

Authors: Yuqi Li, Yuanzhong Zheng, Zhongtian Guo, Yaoxuan Wang, Jianjun Yin, Haojun Fei

Abstract: This paper presents SpecWav-Attack, an adversarial model for detecting speakers in anonymized speech. It leverages Wav2Vec2 for feature extraction and incorporates spectrogram resizing and incremental training for improved performance. Evaluated on librispeech-dev and librispeech-test, SpecWav-Attack outperforms conventional attacks, revealing vulnerabilities in anonymized speech systems and empha… ▽ More This paper presents SpecWav-Attack, an adversarial model for detecting speakers in anonymized speech. It leverages Wav2Vec2 for feature extraction and incorporates spectrogram resizing and incremental training for improved performance. Evaluated on librispeech-dev and librispeech-test, SpecWav-Attack outperforms conventional attacks, revealing vulnerabilities in anonymized speech systems and emphasizing the need for stronger defenses, benchmarked against the ICASSP 2025 Attacker Challenge. △ Less

Submitted 10 January, 2025; originally announced May 2025.

Comments: 2 pages,3 figures,1 chart

MSC Class: I.2.0

Model-free Online Learning for the Kalman Filter: Forgetting Factor and Logarithmic Regret

Authors: Jiachen Qian, Yang Zheng

Abstract: We consider the problem of online prediction for an unknown, non-explosive linear stochastic system. With a known system model, the optimal predictor is the celebrated Kalman filter. In the case of unknown systems, existing approaches based on recursive least squares and its variants may suffer from degraded performance due to the highly imbalanced nature of the regression model. This imbalance ca… ▽ More We consider the problem of online prediction for an unknown, non-explosive linear stochastic system. With a known system model, the optimal predictor is the celebrated Kalman filter. In the case of unknown systems, existing approaches based on recursive least squares and its variants may suffer from degraded performance due to the highly imbalanced nature of the regression model. This imbalance can easily lead to overfitting and thus degrade prediction accuracy. We tackle this problem by injecting an inductive bias into the regression model via {exponential forgetting}. While exponential forgetting is a common wisdom in online learning, it is typically used for re-weighting data. In contrast, our approach focuses on balancing the regression model. This achieves a better trade-off between {regression} and {regularization errors}, and simultaneously reduces the {accumulation error}. With new proof techniques, we also provide a sharper logarithmic regret bound of $O(\log^3 N)$, where $N$ is the number of observations. △ Less

Submitted 13 May, 2025; originally announced May 2025.

Predicting Diabetic Macular Edema Treatment Responses Using OCT: Dataset and Methods of APTOS Competition

Authors: Weiyi Zhang, Peranut Chotcomwongse, Yinwen Li, Pusheng Xu, Ruijie Yao, Lianhao Zhou, Yuxuan Zhou, Hui Feng, Qiping Zhou, Xinyue Wang, Shoujin Huang, Zihao Jin, Florence H. T. Chung, Shujun Wang, Yalin Zheng, Mingguang He, Danli Shi, Paisan Ruamviboonsuk

Abstract: Diabetic macular edema (DME) significantly contributes to visual impairment in diabetic patients. Treatment responses to intravitreal therapies vary, highlighting the need for patient stratification to predict therapeutic benefits and enable personalized strategies. To our knowledge, this study is the first to explore pre-treatment stratification for predicting DME treatment responses. To advance… ▽ More Diabetic macular edema (DME) significantly contributes to visual impairment in diabetic patients. Treatment responses to intravitreal therapies vary, highlighting the need for patient stratification to predict therapeutic benefits and enable personalized strategies. To our knowledge, this study is the first to explore pre-treatment stratification for predicting DME treatment responses. To advance this research, we organized the 2nd Asia-Pacific Tele-Ophthalmology Society (APTOS) Big Data Competition in 2021. The competition focused on improving predictive accuracy for anti-VEGF therapy responses using ophthalmic OCT images. We provided a dataset containing tens of thousands of OCT images from 2,000 patients with labels across four sub-tasks. This paper details the competition's structure, dataset, leading methods, and evaluation metrics. The competition attracted strong scientific community participation, with 170 teams initially registering and 41 reaching the final round. The top-performing team achieved an AUC of 80.06%, highlighting the potential of AI in personalized DME treatment and clinical decision-making. △ Less

Submitted 9 May, 2025; originally announced May 2025.

Comments: 42 pages,5 tables, 12 figures, challenge report

Rapid diagnostics of reconfigurable intelligent surfaces using space-time-coding modulation

Authors: Yi Ning Zheng, Lei Zhang, Xiao Qing Chen, Marco Rossi, Giuseppe Castaldi, Shuo Liu, Tie Jun Cui, Vincenzo Galdi

Abstract: Reconfigurable intelligent surfaces (RISs) have emerged as a key technology for shaping smart wireless environments in next-generation wireless communication systems. To support the large-scale deployment of RISs, a reliable and efficient diagnostic method is essential to ensure optimal performance. In this work, a robust and efficient approach for RIS diagnostics is proposed using a space-time co… ▽ More Reconfigurable intelligent surfaces (RISs) have emerged as a key technology for shaping smart wireless environments in next-generation wireless communication systems. To support the large-scale deployment of RISs, a reliable and efficient diagnostic method is essential to ensure optimal performance. In this work, a robust and efficient approach for RIS diagnostics is proposed using a space-time coding strategy with orthogonal codes. The method encodes the reflected signals from individual RIS elements into distinct code channels, enabling the recovery of channel power at the receiving terminals for fault identification. Theoretical analysis shows that the normally functioning elements generate high power in their respective code channels, whereas the faulty elements exhibit significantly lower power. This distinction enables rapid and accurate diagnostics of elements' operational states through simple signal processing techniques. Simulation results validate the effectiveness of the proposed method, even under high fault ratios and varying reception angles. Proof-of-principle experiments on two RIS prototypes are conducted, implementing two coding strategies: direct and segmented. Experimental results in a realistic scenario confirm the reliability of the diagnostic method, demonstrating its potential for large-scale RIS deployment in future wireless communication systems and radar applications. △ Less

Submitted 6 May, 2025; originally announced May 2025.

Comments: 30 pages, 6 figures, 1 table, supporting information

Selective Variable Convolution Meets Dynamic Content Guided Attention for Infrared Small Target Detection

Authors: Yirui Chen, Yiming Zhu, Yuxin Jing, Tianpei Zhang, Yuchen Zheng

Abstract: Infrared Small Target Detection (IRSTD) system aims to identify small targets in complex backgrounds. Due to the convolution operation in Convolutional Neural Networks (CNNs), applying traditional CNNs to IRSTD presents challenges, since the feature extraction of small targets is often insufficient, resulting in the loss of critical features. To address these issues, we propose a dynamic content g… ▽ More Infrared Small Target Detection (IRSTD) system aims to identify small targets in complex backgrounds. Due to the convolution operation in Convolutional Neural Networks (CNNs), applying traditional CNNs to IRSTD presents challenges, since the feature extraction of small targets is often insufficient, resulting in the loss of critical features. To address these issues, we propose a dynamic content guided attention multiscale feature aggregation network (DCGANet), which adheres to the attention principle of 'coarse-to-fine' and achieves high detection accuracy. First, we propose a selective variable convolution (SVC) module that integrates the benefits of standard convolution, irregular deformable convolution, and multi-rate dilated convolution. This module is designed to expand the receptive field and enhance non-local features, thereby effectively improving the discrimination of targets from backgrounds. Second, the core component of DCGANet is a two-stage content guided attention module. This module employs two-stage attention mechanism to initially direct the network's focus to salient regions within the feature maps and subsequently determine whether these regions correspond to targets or background interference. By retaining the most significant responses, this mechanism effectively suppresses false alarms. Additionally, we propose adaptive dynamic feature fusion (ADFF) module to substitute for static feature cascading. This dynamic feature fusion strategy enables DCGANet to adaptively integrate contextual features, thereby enhancing its ability to discriminate true targets from false alarms. DCGANet has achieved new benchmarks across multiple datasets. △ Less

Submitted 30 April, 2025; originally announced April 2025.

Make Both Ends Meet: A Synergistic Optimization Infrared Small Target Detection with Streamlined Computational Overhead

Authors: Yuxin Jing, Yuchen Zheng, Jufeng Zhao, Guangmang Cui, Tianpei Zhang

Abstract: Infrared small target detection(IRSTD) is widely recognized as a challenging task due to the inherent limitations of infrared imaging, including low signal-to-noise ratios, lack of texture details, and complex background interference. While most existing methods model IRSTD as a semantic segmentation task, but they suffer from two critical drawbacks: (1)blurred target boundaries caused by long-dis… ▽ More Infrared small target detection(IRSTD) is widely recognized as a challenging task due to the inherent limitations of infrared imaging, including low signal-to-noise ratios, lack of texture details, and complex background interference. While most existing methods model IRSTD as a semantic segmentation task, but they suffer from two critical drawbacks: (1)blurred target boundaries caused by long-distance imaging dispersion; and (2) excessive computational overhead due to indiscriminate feature stackin. To address these issues, we propose the Lightweight Efficiency Infrared Small Target Detection (LE-IRSTD), a lightweight and efficient framework based on YOLOv8n, with following key innovations. Firstly, we identify that the multiple bottleneck structures within the C2f component of the YOLOv8-n backbone contribute to an increased computational burden. Therefore, we implement the Mobile Inverted Bottleneck Convolution block (MBConvblock) and Bottleneck Structure block (BSblock) in the backbone, effectively balancing the trade-off between computational efficiency and the extraction of deep semantic information. Secondly, we introduce the Attention-based Variable Convolution Stem (AVCStem) structure, substituting the final convolution with Variable Kernel Convolution (VKConv), which allows for adaptive convolutional kernels that can transform into various shapes, facilitating the receptive field for the extraction of targets. Finally, we employ Global Shuffle Convolution (GSConv) to shuffle the channel dimension features obtained from different convolutional approaches, thereby enhancing the robustness and generalization capabilities of our method. Experimental results demonstrate that our LE-IRSTD method achieves compelling results in both accuracy and lightweight performance, outperforming several state-of-the-art deep learning methods. △ Less

Submitted 4 May, 2025; v1 submitted 30 April, 2025; originally announced April 2025.

NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results

Authors: Xin Li, Yeying Jin, Xin Jin, Zongwei Wu, Bingchen Li, Yufei Wang, Wenhan Yang, Yu Li, Zhibo Chen, Bihan Wen, Robby T. Tan, Radu Timofte, Qiyu Rong, Hongyuan Jing, Mengmeng Zhang, Jinglong Li, Xiangyu Lu, Yi Ren, Yuting Liu, Meng Zhang, Xiang Chen, Qiyuan Guan, Jiangxin Dong, Jinshan Pan, Conglin Gou , et al. (112 additional authors not shown)

Abstract: This paper reviews the NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images. This challenge received a wide range of impressive solutions, which are developed and evaluated using our collected real-world Raindrop Clarity dataset. Unlike existing deraining datasets, our Raindrop Clarity dataset is more diverse and challenging in degradation types and contents, which includ… ▽ More This paper reviews the NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images. This challenge received a wide range of impressive solutions, which are developed and evaluated using our collected real-world Raindrop Clarity dataset. Unlike existing deraining datasets, our Raindrop Clarity dataset is more diverse and challenging in degradation types and contents, which includes day raindrop-focused, day background-focused, night raindrop-focused, and night background-focused degradations. This dataset is divided into three subsets for competition: 14,139 images for training, 240 images for validation, and 731 images for testing. The primary objective of this challenge is to establish a new and powerful benchmark for the task of removing raindrops under varying lighting and focus conditions. There are a total of 361 participants in the competition, and 32 teams submitting valid solutions and fact sheets for the final testing phase. These submissions achieved state-of-the-art (SOTA) performance on the Raindrop Clarity dataset. The project can be found at https://lixinustc.github.io/CVPR-NTIRE2025-RainDrop-Competition.github.io/. △ Less

Submitted 19 April, 2025; v1 submitted 17 April, 2025; originally announced April 2025.

Comments: Challenge Report of CVPR NTIRE 2025; 26 pages; Methods from 32 teams

Dictionary-free Koopman Predictive Control for Autonomous Vehicles in Mixed Traffic

Authors: Xu Shang, Zhaojian Li, Yang Zheng

Abstract: Koopman Model Predictive Control (KMPC) and Data-EnablEd Predictive Control (DeePC) use linear models to approximate nonlinear systems and integrate them with predictive control. Both approaches have recently demonstrated promising performance in controlling Connected and Autonomous Vehicles (CAVs) in mixed traffic. However, selecting appropriate lifting functions for the Koopman operator in KMPC… ▽ More Koopman Model Predictive Control (KMPC) and Data-EnablEd Predictive Control (DeePC) use linear models to approximate nonlinear systems and integrate them with predictive control. Both approaches have recently demonstrated promising performance in controlling Connected and Autonomous Vehicles (CAVs) in mixed traffic. However, selecting appropriate lifting functions for the Koopman operator in KMPC is challenging, while the data-driven representation from Willems' fundamental lemma in DeePC must be updated to approximate the local linearization when the equilibrium traffic state changes. In this paper, we propose a dictionary-free Koopman model predictive control (DF-KMPC) for CAV control. In particular, we first introduce a behavioral perspective to identify the optimal dictionary-free Koopman linear model. We then utilize an iterative algorithm to compute a data-driven approximation of the dictionary-free Koopman representation. Integrating this data-driven linear representation with predictive control leads to our DF-KMPC, which eliminates the need to select lifting functions and update the traffic equilibrium state. Nonlinear traffic simulations show that DF-KMPC effectively mitigates traffic waves and improves tracking performance. △ Less

Submitted 8 April, 2025; originally announced April 2025.

Aligned Better, Listen Better for Audio-Visual Large Language Models

Authors: Yuxin Guo, Shuailei Ma, Shijie Ma, Xiaoyi Bao, Chen-Wei Xie, Kecheng Zheng, Tingyu Weng, Siyang Sun, Yun Zheng, Wei Zou

Abstract: Audio is essential for multimodal video understanding. On the one hand, video inherently contains audio, which supplies complementary information to vision. Besides, video large language models (Video-LLMs) can encounter many audio-centric settings. However, existing Video-LLMs and Audio-Visual Large Language Models (AV-LLMs) exhibit deficiencies in exploiting audio information, leading to weak un… ▽ More Audio is essential for multimodal video understanding. On the one hand, video inherently contains audio, which supplies complementary information to vision. Besides, video large language models (Video-LLMs) can encounter many audio-centric settings. However, existing Video-LLMs and Audio-Visual Large Language Models (AV-LLMs) exhibit deficiencies in exploiting audio information, leading to weak understanding and hallucinations. To solve the issues, we delve into the model architecture and dataset. (1) From the architectural perspective, we propose a fine-grained AV-LLM, namely Dolphin. The concurrent alignment of audio and visual modalities in both temporal and spatial dimensions ensures a comprehensive and accurate understanding of videos. Specifically, we devise an audio-visual multi-scale adapter for multi-scale information aggregation, which achieves spatial alignment. For temporal alignment, we propose audio-visual interleaved merging. (2) From the dataset perspective, we curate an audio-visual caption and instruction-tuning dataset, called AVU. It comprises 5.2 million diverse, open-ended data tuples (video, audio, question, answer) and introduces a novel data partitioning strategy. Extensive experiments show our model not only achieves remarkable performance in audio-visual understanding, but also mitigates potential hallucinations. △ Less

Submitted 2 April, 2025; originally announced April 2025.

Comments: Accepted to ICLR 2025

Non-Asymptotic Analysis of Classical Spectrum Estimators for $L$-mixing Time-series Data with Unknown Means

Authors: Yuping Zheng, Andrew Lamperski

Abstract: Spectral estimation is an important tool in time series analysis, with applications including economics, astronomy, and climatology. The asymptotic theory for non-parametric estimation is well-known but the development of non-asymptotic theory is still ongoing. Our recent work obtained the first non-asymptotic error bounds on the Bartlett and Welch methods for $L$-mixing stochastic processes. The… ▽ More Spectral estimation is an important tool in time series analysis, with applications including economics, astronomy, and climatology. The asymptotic theory for non-parametric estimation is well-known but the development of non-asymptotic theory is still ongoing. Our recent work obtained the first non-asymptotic error bounds on the Bartlett and Welch methods for $L$-mixing stochastic processes. The class of $L$-mixing processes contains common models in time series analysis, including autoregressive processes and measurements of geometrically ergodic Markov chains. Our prior analysis assumes that the process has zero mean. While zero-mean assumptions are common, real-world time-series data often has unknown, non-zero mean. In this work, we derive non-asymptotic error bounds for both Bartlett and Welch estimators for $L$-mixing time-series data with unknown means. The obtained error bounds are of $O(\frac{1}{\sqrt{k}})$, where $k$ is the number of data segments used in the algorithm, which are tighter than our previous results under the zero-mean assumption. △ Less

Submitted 31 March, 2025; originally announced April 2025.

Comments: 7 pages, 2 figures, Under Review for Conference on Decision and Control 2025

JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization

Authors: Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, Tat-Seng Chua

Abstract: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Built upon the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts. To ensure optimal synchronization, we introduce a fine-grained spatio-temporal alignme… ▽ More This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG). Built upon the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to generate high-quality audio and video content simultaneously from open-ended user prompts. To ensure optimal synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, consisting of 10,140 high-quality text-captioned sounding videos spanning diverse scenes and complex real-world scenarios. Further, we specifically devise a robust metric for evaluating the synchronization between generated audio-video pairs in real-world complex content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and dataset will be made publicly available at https://javisdit.github.io/. △ Less

Submitted 30 March, 2025; originally announced March 2025.

Comments: Work in progress. Homepage: https://javisdit.github.io/

Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations

Authors: Jinming Chen, Jingyi Fang, Yuanzhong Zheng, Yaoxuan Wang, Haojun Fei

Abstract: Emotion recognition plays a pivotal role in intelligent human-machine interaction systems. Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy. However, the lack of high-quality multimodal data and the challenge of achieving optimal alignment between different modalities significantly limit the potential for improvement in multimodal appr… ▽ More Emotion recognition plays a pivotal role in intelligent human-machine interaction systems. Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy. However, the lack of high-quality multimodal data and the challenge of achieving optimal alignment between different modalities significantly limit the potential for improvement in multimodal approaches. In this paper, the proposed Qieemo framework effectively utilizes the pretrained automatic speech recognition (ASR) model backbone which contains naturally frame aligned textual and emotional features, to achieve precise emotion classification solely based on the audio modality. Furthermore, we design the multimodal fusion (MMF) module and cross-modal attention (CMA) module in order to fuse the phonetic posteriorgram (PPG) and emotional features extracted by the ASR encoder for improving recognition accuracy. The experimental results on the IEMOCAP dataset demonstrate that Qieemo outperforms the benchmark unimodal, multimodal, and self-supervised models with absolute improvements of 3.0%, 1.2%, and 1.9% respectively. △ Less

Submitted 5 March, 2025; originally announced March 2025.

One-Point Residual Feedback Algorithms for Distributed Online Convex and Non-convex Optimization

Authors: Yaowen Wang, Lipo Mo, Min Zuo, Yuanshi Zheng

Abstract: This paper mainly addresses the distributed online optimization problem where the local objective functions are assumed to be convex or non-convex. First, the distributed algorithms are proposed for the convex and non-convex situations, where the one-point residual feedback technology is introduced to estimate gradient of local objective functions. Then the regret bounds of the proposed algorithms… ▽ More This paper mainly addresses the distributed online optimization problem where the local objective functions are assumed to be convex or non-convex. First, the distributed algorithms are proposed for the convex and non-convex situations, where the one-point residual feedback technology is introduced to estimate gradient of local objective functions. Then the regret bounds of the proposed algorithms are derived respectively under the assumption that the local objective functions are Lipschitz or smooth, which implies that the regrets are sublinear. Finally, we give two numerical examples of distributed convex optimization and distributed resources allocation problem to illustrate the effectiveness of the proposed algorithm. △ Less

Submitted 21 March, 2025; originally announced March 2025.

Joint Semantic Transmission and Resource Allocation for Intelligent Computation Task Offloading in MEC Systems

Authors: Yuanpeng Zheng, Tiankui Zhang, Xidong Mu, Yuanwei Liu, Rong Huang

Abstract: Mobile edge computing (MEC) enables the provision of high-reliability and low-latency applications by offering computation and storage resources in close proximity to end-users. Different from traditional computation task offloading in MEC systems, the large data volume and complex task computation of artificial intelligence involved intelligent computation task offloading have increased greatly.… ▽ More Mobile edge computing (MEC) enables the provision of high-reliability and low-latency applications by offering computation and storage resources in close proximity to end-users. Different from traditional computation task offloading in MEC systems, the large data volume and complex task computation of artificial intelligence involved intelligent computation task offloading have increased greatly. To address this challenge, we propose a MEC system for multiple base stations and multiple terminals, which exploits semantic transmission and early exit of inference. Based on this, we investigate a joint semantic transmission and resource allocation problem for maximizing system reward combined with analysis of semantic transmission and intelligent computation process. To solve the formulated problem, we decompose it into communication resource allocation subproblem, semantic transmission subproblem, and computation capacity allocation subproblem. Then, we use 3D matching and convex optimization method to solve subproblems based on the block coordinate descent (BCD) framework. The optimized feasible solutions are derived from an efficient BCD based joint semantic transmission and resource allocation algorithm in MEC systems. Our simulation demonstrates that: 1) The proposed algorithm significantly improves the delay performance for MEC systems compared with benchmarks; 2) The design of transmission mode and early exit of inference greatly increases system reward during offloading; and 3) Our proposed system achieves efficient utilization of resources from the perspective of system reward in the intelligent scenario. △ Less

Submitted 10 March, 2025; originally announced March 2025.

Synthesizing Individualized Aging Brains in Health and Disease with Generative Models and Parallel Transport

Authors: Jingru Fu, Yuqi Zheng, Neel Dey, Daniel Ferreira, Rodrigo Moreno

Abstract: Simulating prospective magnetic resonance imaging (MRI) scans from a given individual brain image is challenging, as it requires accounting for canonical changes in aging and/or disease progression while also considering the individual brain's current status and unique characteristics. While current deep generative models can produce high-resolution anatomically accurate templates for population-w… ▽ More Simulating prospective magnetic resonance imaging (MRI) scans from a given individual brain image is challenging, as it requires accounting for canonical changes in aging and/or disease progression while also considering the individual brain's current status and unique characteristics. While current deep generative models can produce high-resolution anatomically accurate templates for population-wide studies, their ability to predict future aging trajectories for individuals remains limited, particularly in capturing subject-specific neuroanatomical variations over time. In this study, we introduce Individualized Brain Synthesis (InBrainSyn), a framework for synthesizing high-resolution subject-specific longitudinal MRI scans that simulate neurodegeneration in both Alzheimer's disease (AD) and normal aging. InBrainSyn uses a parallel transport algorithm to adapt the population-level aging trajectories learned by a generative deep template network, enabling individualized aging synthesis. As InBrainSyn uses diffeomorphic transformations to simulate aging, the synthesized images are topologically consistent with the original anatomy by design. We evaluated InBrainSyn both quantitatively and qualitatively on AD and healthy control cohorts from the Open Access Series of Imaging Studies - version 3 dataset. Experimentally, InBrainSyn can also model neuroanatomical transitions between normal aging and AD. An evaluation of an external set supports its generalizability. Overall, with only a single baseline scan, InBrainSyn synthesizes realistic 3D spatiotemporal T1w MRI scans, producing personalized longitudinal aging trajectories. The code for InBrainSyn is available at: https://github.com/Fjr9516/InBrainSyn. △ Less

Submitted 28 February, 2025; originally announced February 2025.

Comments: 20 pages, 9 figures, 6 tables, diffeomorphic registration, parallel transport, brain aging, medical image generation, Alzheimer's disease

Reachability-Aware Reinforcement Learning for Collision Avoidance in Human-Machine Shared Control

Authors: Shiyue Zhao, Junzhi Zhang, Neda Masoud, Jianxiong Li, Yinan Zheng, Xiaohui Hou

Abstract: Human-machine shared control in critical collision scenarios aims to aid drivers' accident avoidance through intervening only when necessary. Existing methods count on replanning collision-free trajectories and imposing human-machine tracking, which usually interrupts the driver's intent and increases the risk of conflict. Additionally, the lack of guaranteed trajectory feasibility under extreme c… ▽ More Human-machine shared control in critical collision scenarios aims to aid drivers' accident avoidance through intervening only when necessary. Existing methods count on replanning collision-free trajectories and imposing human-machine tracking, which usually interrupts the driver's intent and increases the risk of conflict. Additionally, the lack of guaranteed trajectory feasibility under extreme conditions can compromise safety and reliability. This paper introduces a Reachability-Aware Reinforcement Learning framework for shared control, guided by Hamilton-Jacobi (HJ) reachability analysis. Machine intervention is activated only when the vehicle approaches the Collision Avoidance Reachable Set (CARS), which represents states where collision is unavoidable. First, we precompute the reachability distributions and the CARS by solving the Bellman equation using offline data. To reduce human-machine conflicts, we develop a driver model for sudden obstacles and propose an authority allocation strategy considering key collision avoidance features. Finally, we train a reinforcement learning agent to reduce human-machine conflicts while enforcing the hard constraint of avoiding entry into the CARS. The proposed method was tested on a real vehicle platform. Results show that the controller intervenes effectively near CARS to prevent collisions while maintaining improved original driving task performance. Robustness analysis further supports its flexibility across different driver attributes. △ Less

Submitted 14 February, 2025; originally announced February 2025.

Comments: 12 pages, 13 figures, submitted to Advanced Engineering Informatics (ADVEI)

A Bundle-based Augmented Lagrangian Framework: Algorithm, Convergence, and Primal-dual Principles

Authors: Feng-Yi Liao, Yang Zheng

Abstract: We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a $\textit{single-loop}$ process. Motivated by the proximal bundle method (PBM), we use a $\textit{bundle}$ of past iterates to approximate the subproblem in ALM to get… ▽ More We propose a new bundle-based augmented Lagrangian framework for solving constrained convex problems. Unlike the classical (inexact) augmented Lagrangian method (ALM) that has a nested double-loop structure, our framework features a $\textit{single-loop}$ process. Motivated by the proximal bundle method (PBM), we use a $\textit{bundle}$ of past iterates to approximate the subproblem in ALM to get a computationally efficient update at each iteration. We establish sub-linear convergences for primal feasibility, primal cost values, and dual iterates under mild assumptions. With further regularity conditions, such as quadratic growth, our algorithm enjoys $\textit{linear}$ convergences. Importantly, this linear convergence can happen for a class of conic optimization problems, including semidefinite programs. Our proof techniques leverage deep connections with inexact ALM and primal-dual principles with PBM. △ Less

Submitted 12 February, 2025; originally announced February 2025.

Comments: 36 pages, 4 Figures

Comprehensive Analysis of Thermal Dissipation in Lithium-Ion Battery Packs

Authors: Xuguang Zhang, Hexiang Zhang, Amjad Almansour, Mrityunjay Singh, Hengling Zhu, Michael C. Halbig, Yi Zheng

Abstract: Effective thermal management is critical for lithium-ion battery packs' safe and efficient operations, particularly in applications such as drones, where compact designs and varying airflow conditions present unique challenges. This study investigates the thermal performance of a 16-cell lithium-ion battery pack by optimizing cooling airflow configurations and integrating phase change materials (P… ▽ More Effective thermal management is critical for lithium-ion battery packs' safe and efficient operations, particularly in applications such as drones, where compact designs and varying airflow conditions present unique challenges. This study investigates the thermal performance of a 16-cell lithium-ion battery pack by optimizing cooling airflow configurations and integrating phase change materials (PCMs) for enhanced heat dissipation. Seven geometric configurations were evaluated under airflow speeds ranging from 0 to 15 m/s, reflecting the operational conditions of civilian drones. A comprehensive 3D simulation approach was used to analyze the effects of inlet and outlet configurations, airflow dynamics, and PCM phase transition behavior. Results indicate that the trapezoidal (wide-base) configuration, paired with a 5-inlet and 1-outlet setup, achieves the most balanced performance, effectively maintaining optimal operating temperatures across low and high-speed airflow conditions. PCM integration further stabilized thermal behavior, with phase change durations extending to 12.5 min under tested conditions. These findings highlight the importance of geometric optimization and material integration in advancing compact and reliable thermal management systems for energy-dense battery packs. This study provides a foundation for designing efficient cooling strategies tailored to lightweight applications such as drones and portable energy storage systems. △ Less

Submitted 10 February, 2025; originally announced February 2025.

Comments: 20 pages, five figures, introduced the thermal management of Lithium-ion battery

Synthetic Poisoning Attacks: The Impact of Poisoned MRI Image on U-Net Brain Tumor Segmentation

Authors: Tianhao Li, Tianyu Zeng, Yujia Zheng, Chulong Zhang, Jingyu Lu, Haotian Huang, Chuangxin Chu, Fang-Fang Yin, Zhenyu Yang

Abstract: Deep learning-based medical image segmentation models, such as U-Net, rely on high-quality annotated datasets to achieve accurate predictions. However, the increasing use of generative models for synthetic data augmentation introduces potential risks, particularly in the absence of rigorous quality control. In this paper, we investigate the impact of synthetic MRI data on the robustness and segmen… ▽ More Deep learning-based medical image segmentation models, such as U-Net, rely on high-quality annotated datasets to achieve accurate predictions. However, the increasing use of generative models for synthetic data augmentation introduces potential risks, particularly in the absence of rigorous quality control. In this paper, we investigate the impact of synthetic MRI data on the robustness and segmentation accuracy of U-Net models for brain tumor segmentation. Specifically, we generate synthetic T1-contrast-enhanced (T1-Ce) MRI scans using a GAN-based model with a shared encoding-decoding framework and shortest-path regularization. To quantify the effect of synthetic data contamination, we train U-Net models on progressively "poisoned" datasets, where synthetic data proportions range from 16.67% to 83.33%. Experimental results on a real MRI validation set reveal a significant performance degradation as synthetic data increases, with Dice coefficients dropping from 0.8937 (33.33% synthetic) to 0.7474 (83.33% synthetic). Accuracy and sensitivity exhibit similar downward trends, demonstrating the detrimental effect of synthetic data on segmentation robustness. These findings underscore the importance of quality control in synthetic data integration and highlight the risks of unregulated synthetic augmentation in medical image analysis. Our study provides critical insights for the development of more reliable and trustworthy AI-driven medical imaging systems. △ Less

Submitted 6 February, 2025; originally announced February 2025.

Proxy Prompt: Endowing SAM and SAM 2 with Auto-Interactive-Prompt for Medical Segmentation

Authors: Wang Xinyi, Kang Hongyu, Wei Peishan, Shuai Li, Yu Sun, Sai Kit Lam, Yongping Zheng

Abstract: In this paper, we aim to address the unmet demand for automated prompting and enhanced human-model interactions of SAM and SAM2 for the sake of promoting their widespread clinical adoption. Specifically, we propose Proxy Prompt (PP), auto-generated by leveraging non-target data with a pre-annotated mask. We devise a novel 3-step context-selection strategy for adaptively selecting the most represen… ▽ More In this paper, we aim to address the unmet demand for automated prompting and enhanced human-model interactions of SAM and SAM2 for the sake of promoting their widespread clinical adoption. Specifically, we propose Proxy Prompt (PP), auto-generated by leveraging non-target data with a pre-annotated mask. We devise a novel 3-step context-selection strategy for adaptively selecting the most representative contextual information from non-target data via vision mamba and selective maps, empowering the guiding capability of non-target image-mask pairs for segmentation on target image/video data. To reinforce human-model interactions in PP, we further propose a contextual colorization module via a dual-reverse cross-attention to enhance interactions between target features and contextual-embedding with amplifying distinctive features of user-defined object(s). Via extensive evaluations, our method achieves state-of-the-art performance on four public datasets and yields comparable results with fully-trained models, even when trained with only 16 image masks. △ Less

Submitted 8 May, 2025; v1 submitted 5 February, 2025; originally announced February 2025.

Waste Animal Bone-derived Calcium Phosphate Particles with High Solar Reflectance

Authors: Nathaniel LeCompte, Andrew Caratenuto, Yi Zheng

Abstract: Highly reflective Calcium Phosphate (CAP) nanoparticles have been obtained from waste chicken and porcine bones. Chicken and pork bones have been processed and calcined at temperatures between 600°C and 1200°C to remove organic material and resulting in CAP bio-ceramic compounds with high reflectance. The reflectivity of the materials in the solar wavelength region is on par with chemically synthe… ▽ More Highly reflective Calcium Phosphate (CAP) nanoparticles have been obtained from waste chicken and porcine bones. Chicken and pork bones have been processed and calcined at temperatures between 600°C and 1200°C to remove organic material and resulting in CAP bio-ceramic compounds with high reflectance. The reflectivity of the materials in the solar wavelength region is on par with chemically synthesized CAP. The high reflectivity, consistently over 90%, as well as the size distribution and packing density of the nanoparticles obtained in these early bone studies make a strong case for pursuing this avenue to obtain pigment for high solar reflectivity applications, such as passive daytime radiative cooling. The results presented indicate a viable path toward a cost-effective and eco-friendly source of highly reflective cooling pigments. By sourcing calcium phosphates from animal bones, there is also the potential to divert large quantities of bone waste generated by the meat industry from landfills, further contributing toward sustainability and energy reduction efforts in the construction industry and beyond. △ Less

Submitted 29 January, 2025; originally announced January 2025.

Comments: 15 pages, 4 figures

FreeCodec: A disentangled neural speech codec with fewer tokens

Authors: Youqiang Zheng, Weiping Tu, Yueteng Kang, Jie Chen, Yike Zhang, Li Xiao, Yuhong Yang, Long Ma

Abstract: Neural speech codecs have gained great attention for their outstanding reconstruction with discrete token representations. It is a crucial component in generative tasks such as speech coding and large language models (LLM). However, most works based on residual vector quantization perform worse with fewer tokens due to low coding efficiency for modeling complex coupled information. In this p… ▽ More Neural speech codecs have gained great attention for their outstanding reconstruction with discrete token representations. It is a crucial component in generative tasks such as speech coding and large language models (LLM). However, most works based on residual vector quantization perform worse with fewer tokens due to low coding efficiency for modeling complex coupled information. In this paper, we propose a neural speech codec named FreeCodec which employs a more effective encoding framework by decomposing intrinsic properties of speech into different components: 1) a global vector is extracted as the timbre information, 2) a prosody encoder with a long stride level is used to model the prosody information, 3) the content information is from a content encoder. Using different training strategies, FreeCodec achieves state-of-the-art performance in reconstruction and disentanglement scenarios. Results from subjective and objective experiments demonstrate that our framework outperforms existing methods. △ Less

Submitted 28 June, 2025; v1 submitted 1 December, 2024; originally announced December 2024.

Comments: 5 pages, 2 figures, 3 tables.Code and Demo page:https://github.com/exercise-book-yq/FreeCodec. Accepted to Interspeech 2025

Multimodal 3D Brain Tumor Segmentation with Adversarial Training and Conditional Random Field

Authors: Lan Jiang, Yuchao Zheng, Miao Yu, Haiqing Zhang, Fatemah Aladwani, Alessandro Perelli

Abstract: Accurate brain tumor segmentation remains a challenging task due to structural complexity and great individual differences of gliomas. Leveraging the pre-eminent detail resilience of CRF and spatial feature extraction capacity of V-net, we propose a multimodal 3D Volume Generative Adversarial Network (3D-vGAN) for precise segmentation. The model utilizes Pseudo-3D for V-net improvement, adds condi… ▽ More Accurate brain tumor segmentation remains a challenging task due to structural complexity and great individual differences of gliomas. Leveraging the pre-eminent detail resilience of CRF and spatial feature extraction capacity of V-net, we propose a multimodal 3D Volume Generative Adversarial Network (3D-vGAN) for precise segmentation. The model utilizes Pseudo-3D for V-net improvement, adds conditional random field after generator and use original image as supplemental guidance. Results, using the BraTS-2018 dataset, show that 3D-vGAN outperforms classical segmentation models, including U-net, Gan, FCN and 3D V-net, reaching specificity over 99.8%. △ Less

Submitted 21 November, 2024; originally announced November 2024.

Comments: 13 pages, 7 figures, Annual Conference on Medical Image Understanding and Analysis (MIUA) 2024

MSC Class: 15-11 ACM Class: I.4.6; I.5.4

Journal ref: Medical Image Understanding and Analysis (MIUA), Lecture Notes in Computer Science, Springer, vol. 14859, 2024

QuadWBG: Generalizable Quadrupedal Whole-Body Grasping

Authors: Jilong Wang, Javokhirbek Rajabov, Chaoyi Xu, Yiming Zheng, He Wang

Abstract: Legged robots with advanced manipulation capabilities have the potential to significantly improve household duties and urban maintenance. Despite considerable progress in developing robust locomotion and precise manipulation methods, seamlessly integrating these into cohesive whole-body control for real-world applications remains challenging. In this paper, we present a modular framework for robus… ▽ More Legged robots with advanced manipulation capabilities have the potential to significantly improve household duties and urban maintenance. Despite considerable progress in developing robust locomotion and precise manipulation methods, seamlessly integrating these into cohesive whole-body control for real-world applications remains challenging. In this paper, we present a modular framework for robust and generalizable whole-body loco-manipulation controller based on a single arm-mounted camera. By using reinforcement learning (RL), we enable a robust low-level policy for command execution over 5 dimensions (5D) and a grasp-aware high-level policy guided by a novel metric, Generalized Oriented Reachability Map (GORM). The proposed system achieves state-of-the-art one-time grasping accuracy of 89% in the real world, including challenging tasks such as grasping transparent objects. Through extensive simulations and real-world experiments, we demonstrate that our system can effectively manage a large workspace, from floor level to above body height, and perform diverse whole-body loco-manipulation tasks. △ Less

Submitted 13 January, 2025; v1 submitted 11 November, 2024; originally announced November 2024.

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

Authors: Feng-Yi Liao, Lijun Ding, Yang Zheng

Abstract: Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates… ▽ More Augmented Lagrangian Methods (ALMs) are widely employed in solving constrained optimizations, and some efficient solvers are developed based on this framework. Under the quadratic growth assumption, it is known that the dual iterates and the Karush-Kuhn-Tucker (KKT) residuals of ALMs applied to semidefinite programs (SDPs) converge linearly. In contrast, the convergence rate of the primal iterates has remained elusive. In this paper, we resolve this challenge by establishing new $\textit{quadratic growth}$ and $\textit{error bound}$ properties for primal and dual SDPs under the strict complementarity condition. Our main results reveal that both primal and dual iterates of the ALMs converge linearly contingent solely upon the assumption of strict complementarity and a bounded solution set. This finding provides a positive answer to an open question regarding the asymptotically linear convergence of the primal iterates of ALMs applied to semidefinite optimization. △ Less

Submitted 30 October, 2024; originally announced October 2024.

Comments: 32 pages, 5 figures

A Radio Map Approach for Reduced Pilot CSI Tracking in Massive MIMO Networks

Authors: Yuanshuai Zheng, Junting Chen

Abstract: Massive multiple-input multiple-output (MIMO) systems offer significant potential to enhance wireless communication performance, yet accurate and timely channel state information (CSI) acquisition remains a key challenge. Existing works on CSI estimation and radio map applications typically rely on stationary CSI statistics and accurate location labels. However, the CSI process can be discontinuou… ▽ More Massive multiple-input multiple-output (MIMO) systems offer significant potential to enhance wireless communication performance, yet accurate and timely channel state information (CSI) acquisition remains a key challenge. Existing works on CSI estimation and radio map applications typically rely on stationary CSI statistics and accurate location labels. However, the CSI process can be discontinuous due to user mobility and environmental variations, and inaccurate location data can degrade the performance. By contrast, this paper studies radio-map-embedded CSI tracking and radio map construction without the assumptions of stationary CSI statistics and precise location labels. Using radio maps as the prior information, this paper develops a radio-map-embedded switching Kalman filter (SKF) framework that jointly tracks the location and the CSI with adaptive beamforming for sparse CSI observations under reduced pilots. For radio map construction without precise location labels, the location sequence and the channel covariance matrices are jointly estimated based on a Hidden Markov Model (HMM). An unbiased estimator on the channel covariance matrix is found. Numerical results on ray-traced MIMO channel datasets demonstrate that using 1 pilot in every 10 milliseconds, an average of over 97% of capacity over that of perfect CSI can be achieved, while a conventional Kalman filter (KF) can only achieve 76%. Furthermore, the proposed radio-map-embedded CSI model can reduce the localization error from 30 meters from the prior to 6 meters for radio map construction. △ Less

Submitted 22 June, 2025; v1 submitted 8 October, 2024; originally announced October 2024.

Resource Allocation Based on Optimal Transport Theory in ISAC-Enabled Multi-UAV Networks

Authors: Yufeng Zheng, Lixin Li, Wensheng Lin, Wei Liang, Qinghe Du, Zhu Han

Abstract: This paper investigates the resource allocation optimization for cooperative communication with non-cooperative localization in integrated sensing and communications (ISAC)-enabled multi-unmanned aerial vehicle (UAV) cooperative networks. Our goal is to maximize the weighted sum of the system's average sum rate and the localization quality of service (QoS) by jointly optimizing cell association, c… ▽ More This paper investigates the resource allocation optimization for cooperative communication with non-cooperative localization in integrated sensing and communications (ISAC)-enabled multi-unmanned aerial vehicle (UAV) cooperative networks. Our goal is to maximize the weighted sum of the system's average sum rate and the localization quality of service (QoS) by jointly optimizing cell association, communication power allocation, and sensing power allocation. Since the formulated problem is a mixed-integer nonconvex problem, we propose the alternating iteration algorithm based on optimal transport theory (AIBOT) to solve the optimization problem more effectively. Simulation results demonstrate that the AIBOT can improve the system sum rate by nearly 12% and reduce the localization Cr'amer-Rao bound (CRB) by almost 29% compared to benchmark algorithms. △ Less

Submitted 2 October, 2024; originally announced October 2024.

Morphological-consistent Diffusion Network for Ultrasound Coronal Image Enhancement

Authors: Yihao Zhou, Zixun Huang, Timothy Tin-Yan Lee, Chonglin Wu, Kelly Ka-Lee Lai, De Yang, Alec Lik-hang Hung, Jack Chun-Yiu Cheng, Tsz-Ping Lam, Yong-ping Zheng

Abstract: Ultrasound curve angle (UCA) measurement provides a radiation-free and reliable evaluation for scoliosis based on ultrasound imaging. However, degraded image quality, especially in difficult-to-image patients, can prevent clinical experts from making confident measurements, even leading to misdiagnosis. In this paper, we propose a multi-stage image enhancement framework that models high-quality im… ▽ More Ultrasound curve angle (UCA) measurement provides a radiation-free and reliable evaluation for scoliosis based on ultrasound imaging. However, degraded image quality, especially in difficult-to-image patients, can prevent clinical experts from making confident measurements, even leading to misdiagnosis. In this paper, we propose a multi-stage image enhancement framework that models high-quality image distribution via a diffusion-based model. Specifically, we integrate the underlying morphological information from images taken at different depths of the 3D volume to calibrate the reverse process toward high-quality and high-fidelity image generation. This is achieved through a fusion operation with a learnable tuner module that learns the multi-to-one mapping from multi-depth to high-quality images. Moreover, the separate learning of the high-quality image distribution and the spinal features guarantees the preservation of consistent spinal pose descriptions in the generated images, which is crucial in evaluating spinal deformities. Remarkably, our proposed enhancement algorithm significantly outperforms other enhancement-based methods on ultrasound images in terms of image quality. Ultimately, we conduct the intra-rater and inter-rater measurements of UCA and higher ICC (0.91 and 0.89 for thoracic and lumbar angles) on enhanced images, indicating our method facilitates the measurement of ultrasound curve angles and offers promising prospects for automated scoliosis diagnosis. △ Less

Submitted 25 September, 2024; originally announced September 2024.

Willems' Fundamental Lemma for Nonlinear Systems with Koopman Linear Embedding

Authors: Xu Shang, Jorge Cortés, Yang Zheng

Abstract: Koopman operator theory and Willems' fundamental lemma both can provide (approximated) data-driven linear representation for nonlinear systems. However, choosing lifting functions for the Koopman operator is challenging, and the quality of the data-driven model from Willems' fundamental lemma has no guarantee for general nonlinear systems. In this paper, we extend Willems' fundamental lemma for a… ▽ More Koopman operator theory and Willems' fundamental lemma both can provide (approximated) data-driven linear representation for nonlinear systems. However, choosing lifting functions for the Koopman operator is challenging, and the quality of the data-driven model from Willems' fundamental lemma has no guarantee for general nonlinear systems. In this paper, we extend Willems' fundamental lemma for a class of nonlinear systems that admit a Koopman linear embedding. We first characterize the relationship between the trajectory space of a nonlinear system and that of its Koopman linear embedding. We then prove that the trajectory space of Koopman linear embedding can be formed by a linear combination of rich-enough trajectories from the nonlinear system. Combining these two results leads to a data-driven representation of the nonlinear system, which bypasses the need for the lifting functions and thus eliminates the associated bias errors. Our results illustrate that both the width (more trajectories) and depth (longer trajectories) of the trajectory library are important to ensure the accuracy of the data-driven model. △ Less

Submitted 23 November, 2024; v1 submitted 24 September, 2024; originally announced September 2024.

Unsupervised Attention-Based Multi-Source Domain Adaptation Framework for Drift Compensation in Electronic Nose Systems

Authors: Wenwen Zhang, Shuhao Hu, Zhengyuan Zhang, Yuanjin Zheng, Qi Jie Wang, Zhiping Lin

Abstract: Continuous, long-term monitoring of hazardous, noxious, explosive, and flammable gases in industrial environments using electronic nose (E-nose) systems faces the significant challenge of reduced gas identification accuracy due to time-varying drift in gas sensors. To address this issue, we propose a novel unsupervised attention-based multi-source domain shared-private feature fusion adaptation (A… ▽ More Continuous, long-term monitoring of hazardous, noxious, explosive, and flammable gases in industrial environments using electronic nose (E-nose) systems faces the significant challenge of reduced gas identification accuracy due to time-varying drift in gas sensors. To address this issue, we propose a novel unsupervised attention-based multi-source domain shared-private feature fusion adaptation (AMDS-PFFA) framework for gas identification with drift compensation in E-nose systems. The AMDS-PFFA model effectively leverages labeled data from multiple source domains collected during the initial stage to accurately identify gases in unlabeled gas sensor array drift signals from the target domain. To validate the model's effectiveness, extensive experimental evaluations were conducted using both the University of California, Irvine (UCI) standard drift gas dataset, collected over 36 months, and drift signal data from our self-developed E-nose system, spanning 30 months. Compared to recent drift compensation methods, the AMDS-PFFA model achieves the highest average gas recognition accuracy with strong convergence, attaining 83.20% on the UCI dataset and 93.96% on data from our self-developed E-nose system across all target domain batches. These results demonstrate the superior performance of the AMDS-PFFA model in gas identification with drift compensation, significantly outperforming existing methods. △ Less

Submitted 19 September, 2024; originally announced September 2024.

SafeEar: Content Privacy-Preserving Audio Deepfake Detection

Authors: Xinfeng Li, Kai Li, Yifan Zheng, Chen Yan, Xiaoyu Ji, Wenyuan Xu

Abstract: Text-to-Speech (TTS) and Voice Conversion (VC) models have exhibited remarkable performance in generating realistic and natural audio. However, their dark side, audio deepfake poses a significant threat to both society and individuals. Existing countermeasures largely focus on determining the genuineness of speech based on complete original audio recordings, which however often contain private con… ▽ More Text-to-Speech (TTS) and Voice Conversion (VC) models have exhibited remarkable performance in generating realistic and natural audio. However, their dark side, audio deepfake poses a significant threat to both society and individuals. Existing countermeasures largely focus on determining the genuineness of speech based on complete original audio recordings, which however often contain private content. This oversight may refrain deepfake detection from many applications, particularly in scenarios involving sensitive information like business secrets. In this paper, we propose SafeEar, a novel framework that aims to detect deepfake audios without relying on accessing the speech content within. Our key idea is to devise a neural audio codec into a novel decoupling model that well separates the semantic and acoustic information from audio samples, and only use the acoustic information (e.g., prosody and timbre) for deepfake detection. In this way, no semantic content will be exposed to the detector. To overcome the challenge of identifying diverse deepfake audio without semantic clues, we enhance our deepfake detector with real-world codec augmentation. Extensive experiments conducted on four benchmark datasets demonstrate SafeEar's effectiveness in detecting various deepfake techniques with an equal error rate (EER) down to 2.02%. Simultaneously, it shields five-language speech content from being deciphered by both machine and human auditory analysis, demonstrated by word error rates (WERs) all above 93.93% and our user study. Furthermore, our benchmark constructed for anti-deepfake and anti-content recovery evaluation helps provide a basis for future research in the realms of audio privacy preservation and deepfake detection. △ Less

Submitted 13 September, 2024; originally announced September 2024.

Comments: Accepted by ACM CCS 2024. Please cite this paper as "Xinfeng Li, Kai Li, Yifan Zheng, Chen Yan, Xiaoyu Ji, Wenyuan Xu. SafeEar: Content Privacy-Preserving Audio Deepfake Detection. In Proceedings of ACM Conference on Computer and Communications Security (CCS), 2024."

Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance

Authors: Weiyi Zhang, Siyu Huang, Jiancheng Yang, Ruoyu Chen, Zongyuan Ge, Yingfeng Zheng, Danli Shi, Mingguang He

Abstract: Fundus Fluorescein Angiography (FFA) is a critical tool for assessing retinal vascular dynamics and aiding in the diagnosis of eye diseases. However, its invasive nature and less accessibility compared to Color Fundus (CF) images pose significant challenges. Current CF to FFA translation methods are limited to static generation. In this work, we pioneer dynamic FFA video generation from static CF… ▽ More Fundus Fluorescein Angiography (FFA) is a critical tool for assessing retinal vascular dynamics and aiding in the diagnosis of eye diseases. However, its invasive nature and less accessibility compared to Color Fundus (CF) images pose significant challenges. Current CF to FFA translation methods are limited to static generation. In this work, we pioneer dynamic FFA video generation from static CF images. We introduce an autoregressive GAN for smooth, memory-saving frame-by-frame FFA synthesis. To enhance the focus on dynamic lesion changes in FFA regions, we design a knowledge mask based on clinical experience. Leveraging this mask, our approach integrates innovative knowledge mask-guided techniques, including knowledge-boosted attention, knowledge-aware discriminators, and mask-enhanced patchNCE loss, aimed at refining generation in critical areas and addressing the pixel misalignment challenge. Our method achieves the best FVD of 1503.21 and PSNR of 11.81 compared to other common video generation approaches. Human assessment by an ophthalmologist confirms its high generation quality. Notably, our knowledge mask surpasses supervised lesion segmentation masks, offering a promising non-invasive alternative to traditional FFA for research and clinical applications. The code is available at https://github.com/Michi-3000/Fundus2Video. △ Less

Submitted 27 August, 2024; originally announced August 2024.

Comments: The paper has been accepted by Medical Image Computing and Computer Assisted Intervention Society (MICCAI) 2024

Active Search for Low-altitude UAV Sensing and Communication for Users at Unknown Locations

Authors: Yuanshuai Zheng, Junting Chen

Abstract: This paper studies optimal unmanned aerial vehicle (UAV) placement to ensure line-of-sight (LOS) communication and sensing for a cluster of ground users possibly in deep shadow, while the UAV maintains backhaul connectivity with a base station (BS). The key challenges include unknown user locations, uncertain channel model parameters, and unavailable urban structure. Addressing these challenges, t… ▽ More This paper studies optimal unmanned aerial vehicle (UAV) placement to ensure line-of-sight (LOS) communication and sensing for a cluster of ground users possibly in deep shadow, while the UAV maintains backhaul connectivity with a base station (BS). The key challenges include unknown user locations, uncertain channel model parameters, and unavailable urban structure. Addressing these challenges, this paper focuses on developing an efficient online search strategy which jointly estimates channels, guides UAV positioning, and optimizes resource allocation. Analytically exploiting the geometric properties of the equipotential surface, this paper develops an LOS discovery trajectory on the equipotential surface while the closed-form search directions are determined using perturbation theory. Since the explicit expression of the equipotential surface is not available, this paper proposes to locally construct a channel model for each user in the LOS regime utilizing polynomial regression without depending on user locations or propagation distance. A class of spiral trajectories to simultaneously construct the LOS channels and search on the equipotential surface is developed. An optimal radius of the spiral and an optimal measurement pattern for channel gain estimation are derived to minimize the mean squared error (MSE) of the locally constructed channel. Numerical results on real 3D city maps demonstrate that the proposed scheme achieves over 94% of the performance of a 3D exhaustive search scheme with just a 3-kilometer search. △ Less

Submitted 1 September, 2024; v1 submitted 26 August, 2024; originally announced August 2024.

Hardware Architecture Design of Model-Based Image Reconstruction Towards Palm-size Photoacoustic Tomography

Authors: Yuwei Zheng, Zijian Gao, Yuting Shen, Jiadong Zhang, Daohuai Jiang, Fengyu Liu, Feng Gao, Fei Gao

Abstract: Photoacoustic (PA) imaging technology combines the advantages of optical imaging and ultrasound imaging, showing great potential in biomedical applications. Many preclinical studies and clinical applications urgently require fast, high-quality, low-cost and portable imaging system. Translating advanced image reconstruction algorithms into hardware implementations is highly desired. However, existi… ▽ More Photoacoustic (PA) imaging technology combines the advantages of optical imaging and ultrasound imaging, showing great potential in biomedical applications. Many preclinical studies and clinical applications urgently require fast, high-quality, low-cost and portable imaging system. Translating advanced image reconstruction algorithms into hardware implementations is highly desired. However, existing iterative PA image reconstructions, although exhibit higher accuracy than delay-and-sum algorithm, suffer from high computational cost. In this paper, we introduce a model-based hardware acceleration architecture based on superposed Wave (s-Wave) for palm-size PA tomography (palm-PAT), aiming at enhancing both the speed and performance of image reconstruction at a much lower system cost. To achieve this, we propose an innovative data reuse method that significantly reduces hardware storage resource consumption. We conducted experiments by FPGA implementation of the algorithm, using both phantoms and in vivo human finger data to verify the feasibility of the proposed method. The results demonstrate that our proposed architecture can substantially reduce system cost while maintaining high imaging performance. The hardware-accelerated implementation of the model-based algorithm achieves a speedup of up to approximately 270 times compared to the CPU, while the corresponding energy efficiency ratio is improved by more than 2700 times. △ Less

Submitted 12 August, 2024; originally announced August 2024.

Comments: 11 pages, 13 figures

Semantic Communications with Explicit Semantic Bases: Model, Architecture, and Open Problems

Authors: Fengyu Wang, Yuan Zheng, Wenjun Xu, Junxiao Liang, Ping Zhang

Abstract: The increasing demands for massive data transmission pose great challenges to communication systems. Compared to traditional communication systems that focus on the accurate reconstruction of bit sequences, semantic communications (SemComs), which aim to successfully deliver information connotation, have been regarded as the key technology for next-generation communication systems. Most current Se… ▽ More The increasing demands for massive data transmission pose great challenges to communication systems. Compared to traditional communication systems that focus on the accurate reconstruction of bit sequences, semantic communications (SemComs), which aim to successfully deliver information connotation, have been regarded as the key technology for next-generation communication systems. Most current SemCom systems focus on an E2E trained neural network (NN) for semantic extraction and interpretation, regarding the parameters of the NN as the implicit synchronized background knowledge. However, the implicit knowledge base (KB)-based architectures lack interpretability and flexibility, which limits the performance of SemComs. In this article, we propose a SemCom architecture that employs explicit semantic bases (Sebs), which serve as the basic units to describe semantic information. In specific, the mathematical model of Sebs is first proposed to build explicit KB. Then, the Seb-based SemCom architecture is proposed, consisting of a communication mode and a KB update mode to enable the evolution of communication systems. Specifically, the modules of Sem-codec and channel codec are dedicatedly designed, with the assistance of explicit KB for efficient and robust transmission of semantics. Moreover, unequal error protection is strategically implemented, considering the intent of communications and the importance of Sebs, thereby ensuring reliability of critical semantics. To assess the effectiveness of the proposed Seb-based SemCom architecture, a case study focusing on an image transmission task is conducted. Simulations show that the proposed Seb-based SemComs outperforms state-of-art works in LPIPS by over 20% under varying communication intents, with more robust performance under fluctuating channel conditions, indicating the flexible and robust transmission of the proposed Seb-based SemComs. △ Less

Submitted 10 August, 2024; originally announced August 2024.

SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

Authors: Youqiang Zheng, Weiping Tu, Li Xiao, Xinmeng Xu

Abstract: Neural speech coding is a rapidly developing topic, where state-of-the-art approaches now exhibit superior compression performance than conventional methods. Despite significant progress, existing methods still have limitations in preserving and reconstructing fine details for optimal reconstruction, especially at low bitrates. In this study, we introduce SuperCodec, a neural speech codec that ach… ▽ More Neural speech coding is a rapidly developing topic, where state-of-the-art approaches now exhibit superior compression performance than conventional methods. Despite significant progress, existing methods still have limitations in preserving and reconstructing fine details for optimal reconstruction, especially at low bitrates. In this study, we introduce SuperCodec, a neural speech codec that achieves state-of-the-art performance at low bitrates. It employs a novel back projection method with selective feature fusion for augmented representation. Specifically, we propose to use Selective Up-sampling Back Projection (SUBP) and Selective Down-sampling Back Projection (SDBP) modules to replace the standard up- and down-sampling layers at the encoder and decoder, respectively. Experimental results show that our method outperforms the existing neural speech codecs operating at various bitrates. Specifically, our proposed method can achieve higher quality reconstructed speech at 1 kbps than Lyra V2 at 3.2 kbps and Encodec at 6 kbps. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Comments: Accepted by ICASSP 2024

ProRuka: A highly efficient HMI algorithm for controlling a novel prosthetic hand with 6-DOF using sonomyography

Authors: Vaheh Nazari, Yong-Ping Zheng

Abstract: Sonomyography (SMG) is a novel human-machine interface that controls upper-limb prostheses by monitoring forearm muscle activity using ultrasonic imaging. SMG has been investigated for controlling upper-limb prostheses during the last two decades. The results show that this method, in combination with artificial intelligence, can classify different hand gestures with an accuracy of more than 90%,… ▽ More Sonomyography (SMG) is a novel human-machine interface that controls upper-limb prostheses by monitoring forearm muscle activity using ultrasonic imaging. SMG has been investigated for controlling upper-limb prostheses during the last two decades. The results show that this method, in combination with artificial intelligence, can classify different hand gestures with an accuracy of more than 90%, making it a great alternative control system compared to electromyography (EMG). However, up to now there are few reports of a system integrating SMG together with a prosthesis for testing on amputee subjects to demonstrate its capability in relation to daily activities. In this study, we developed ProRuka, a novel low-cost 6-degree-of-freedom prosthetic hand integrated with the control provided by a SMG system with a wearable ultrasound imaging probe. The classification of hand gestures using different machine learning classification/regression algorithms including KNN, nearest neighbor regression, random forest, decision tree classifier, decision tree regression, support vector regression and support vector machine in combination with a transfer learning model (VGG16) was first evaluated off-line to determine its reliability and precision. Additionally, the developed controlling system were evaluated on two amputees, in real-time experiments using a variety of hand function test kits. The results from an off-line study including ten healthy participants indicated that nine different hand motions can be classified with a success rate of 100%. In addition, the hand function test in real time (using 4 different hand gestures) confirmed that the designed prosthesis with the SMG controlling system can assist amputees to perform a variety of hand movements needed in daily activities. △ Less

Submitted 29 July, 2024; originally announced July 2024.

A Holistic Optimization Framework for Energy Efficient UAV-assisted Fog Computing: Attitude Control, Trajectory Planning and Task Assignment

Authors: Shuaijun Liu, Jinqiu Du, Yaxin Zheng, Jiaying Yin, Yuhui Deng, Jingjin Wu

Abstract: Unmanned Aerial Vehicles (UAVs) have significantly enhanced fog computing by acting as both flexible computation platforms and communication mobile relays. In this paper, we propose a holistic framework that jointly optimizes the total latency and energy consumption for UAV-assisted fog computing in a three-dimensional spatial domain with varying terrain elevations and dynamic task generations. Ou… ▽ More Unmanned Aerial Vehicles (UAVs) have significantly enhanced fog computing by acting as both flexible computation platforms and communication mobile relays. In this paper, we propose a holistic framework that jointly optimizes the total latency and energy consumption for UAV-assisted fog computing in a three-dimensional spatial domain with varying terrain elevations and dynamic task generations. Our proposed framework considers three important and interdependent modules: attitude control, trajectory planning, and task assignment. We first establish a fuzzy proportional-integral-derivative control model to determine the UAV's attitude. Then, we propose an enhanced Ant Colony System (ACS) based algorithm, that includes a safety value and a decoupling mechanism to overcome the convergence issue in classical ACS, to compute the optimal UAV trajectory. Finally, we design an algorithm based on the Particle Swarm Optimization technique, to determine where each offloaded task should be executed. Under our proposed framework, the outcome of one module would affect the decision-making in one other, providing a holistic perspective of the system and thus leading to improved solutions. We demonstrate by extensive simulation results that our proposed framework can significantly improve the overall performance, measured by latency and energy consumption, compared to existing baseline approaches. △ Less

Submitted 5 August, 2024; v1 submitted 20 July, 2024; originally announced July 2024.

Comments: 14 pages, 10 figures

Latent Space Imaging

Authors: Matheus Souza, Yidan Zheng, Kaizhang Kang, Yogeshwar Nath Mishra, Qiang Fu, Wolfgang Heidrich

Abstract: Digital imaging systems have traditionally relied on brute-force measurement and processing of pixels arranged on regular grids. In contrast, the human visual system performs significant data reduction from the large number of photoreceptors to the optic nerve, effectively encoding visual information into a low-bandwidth latent space representation optimized for brain processing. Inspired by this,… ▽ More Digital imaging systems have traditionally relied on brute-force measurement and processing of pixels arranged on regular grids. In contrast, the human visual system performs significant data reduction from the large number of photoreceptors to the optic nerve, effectively encoding visual information into a low-bandwidth latent space representation optimized for brain processing. Inspired by this, we propose a similar approach to advance artificial vision systems. Latent Space Imaging introduces a new paradigm that combines optics and software to encode image information directly into the semantically rich latent space of a generative model. This approach substantially reduces bandwidth and memory demands during image capture and enables a range of downstream tasks focused on the latent space. We validate this principle through an initial hardware prototype based on a single-pixel camera. By implementing an amplitude modulation scheme that encodes into the generative model's latent space, we achieve compression ratios ranging from 1:100 to 1:1000 during imaging, and up to 1:16384 for downstream applications. This approach leverages the model's intrinsic linear boundaries, demonstrating the potential of latent space imaging for highly efficient imaging hardware, adaptable future applications in high-speed imaging, and task-specific cameras with significantly reduced hardware complexity. △ Less

Submitted 23 March, 2025; v1 submitted 9 July, 2024; originally announced July 2024.

Comments: Accepted to CVPR 2025; see http://github.com/vccimaging/latent-imaging

Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition

Authors: Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, Lu Gao, Yi Guo, Minglun Han, Ting Han, Wenchao Hu, Xinying Hu, Yuxiang Hu, Deyu Hua, Lu Huang, Mingkun Huang, Youjia Huang, Jishuo Jin, Fanliu Kong, Zongwei Lan, Tianyu Li , et al. (30 additional authors not shown)

Abstract: Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this wor… ▽ More Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this work, we introduce Seed-ASR, a large language model (LLM) based speech recognition model. Seed-ASR is developed based on the framework of audio conditioned LLM (AcLLM), leveraging the capabilities of LLMs by inputting continuous speech representations together with contextual information into the LLM. Through stage-wise large-scale training and the elicitation of context-aware capabilities in LLM, Seed-ASR demonstrates significant improvement over end-to-end models on comprehensive evaluation sets, including multiple domains, accents/dialects and languages. Additionally, Seed-ASR can be further deployed to support specific needs in various scenarios without requiring extra language models. Compared to recently released large ASR models, Seed-ASR achieves 10%-40% reduction in word (or character, for Chinese) error rates on Chinese and English public test sets, further demonstrating its powerful performance. △ Less

Submitted 10 July, 2024; v1 submitted 5 July, 2024; originally announced July 2024.

Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition

Authors: Jinming Chen, Jingyi Fang, Yuanzhong Zheng, Yaoxuan Wang, Haojun Fei

Abstract: Currently, end-to-end (E2E) speech recognition methods have achieved promising performance. However, auto speech recognition (ASR) models still face challenges in recognizing multi-accent speech accurately. We propose a layer-adapted fusion (LAF) model, called Qifusion-Net, which does not require any prior knowledge about the target accent. Based on dynamic chunk strategy, our approach enables str… ▽ More Currently, end-to-end (E2E) speech recognition methods have achieved promising performance. However, auto speech recognition (ASR) models still face challenges in recognizing multi-accent speech accurately. We propose a layer-adapted fusion (LAF) model, called Qifusion-Net, which does not require any prior knowledge about the target accent. Based on dynamic chunk strategy, our approach enables streaming decoding and can extract frame-level acoustic feature, facilitating fine-grained information fusion. Experiment results demonstrate that our proposed methods outperform the baseline with relative reductions of 22.1$\%$ and 17.2$\%$ in character error rate (CER) across multi accent test datasets on KeSpeech and MagicData-RMAC. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: accpeted by interspeech 2014, 5 pages, 1 figure

Detecting and Identifying Selection Structure in Sequential Data

Authors: Yujia Zheng, Zeyu Tang, Yiwen Qiu, Bernhard Schölkopf, Kun Zhang

Abstract: We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. Since this selection process often distorts statistical analysis, previous work primarily views it as a bias to be corrected and proposes various methods to mitigate its effect. However, while controlling this bias is crucial, selection also offers an opportun… ▽ More We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. Since this selection process often distorts statistical analysis, previous work primarily views it as a bias to be corrected and proposes various methods to mitigate its effect. However, while controlling this bias is crucial, selection also offers an opportunity to provide a deeper insight into the hidden generation process, as it is a fundamental mechanism underlying what we observe. In particular, overlooking selection in sequential data can lead to an incomplete or overcomplicated inductive bias in modeling, such as assuming a universal autoregressive structure for all dependencies. Therefore, rather than merely viewing it as a bias, we explore the causal structure of selection in sequential data to delve deeper into the complete causal process. Specifically, we show that selection structure is identifiable without any parametric assumptions or interventional experiments. Moreover, even in cases where selection variables coexist with latent confounders, we still establish the nonparametric identifiability under appropriate structural conditions. Meanwhile, we also propose a provably correct algorithm to detect and identify selection structures as well as other types of dependencies. The framework has been validated empirically on both synthetic data and real-world music. △ Less

Submitted 29 June, 2024; originally announced July 2024.

Comments: ICML 2024

Scalable Near-Field Localization Based on Partitioned Large-Scale Antenna Array

Authors: Xiaojun Yuan, Yuqing Zheng, Mingchen Zhang, Boyu Teng, Wenjun Jiang

Abstract: This paper studies a passive localization system, where an extremely large-scale antenna array (ELAA) is deployed at the base station (BS) to locate a user equipment (UE) residing in its near-field (Fresnel) region. We propose a novel algorithm, named array partitioning-based location estimation (APLE), for scalable near-field localization. The APLE algorithm is developed based on the basic assump… ▽ More This paper studies a passive localization system, where an extremely large-scale antenna array (ELAA) is deployed at the base station (BS) to locate a user equipment (UE) residing in its near-field (Fresnel) region. We propose a novel algorithm, named array partitioning-based location estimation (APLE), for scalable near-field localization. The APLE algorithm is developed based on the basic assumption that, by partitioning the ELAA into multiple subarrays, the UE can be approximated as in the far-field region of each subarray. We establish a Bayeian inference framework based on the geometric constraints between the UE location and the angles of arrivals (AoAs) at different subarrays. Then, the APLE algorithm is designed based on the message-passing principle for the localization of the UE. APLE exhibits linear computational complexity with the number of BS antennas, leading to a significant reduction in complexity compared to existing methods. We further propose an enhanced APLE (E-APLE) algorithm that refines the location estimate obtained from APLE by following the maximum likelihood principle. The E-APLE algorithm achieves superior localization accuracy compared to APLE while maintaining a linear complexity with the number of BS antennas. Numerical results demonstrate that the proposed APLE and E-APLE algorithms outperform the existing baselines in terms of localization accuracy. △ Less

Submitted 13 May, 2024; originally announced June 2024.

Comments: arXiv admin note: text overlap with arXiv:2312.12342