-
Performance Analysis of Cooperative Integrated Sensing and Communications for 6G Networks
Authors:
Dongsheng Sui,
Cunhua Pan,
Hong Ren,
Jiahua Wan,
Liuchang Zhuo,
Jing Jin,
Qixing Wang,
Jiangzhou Wang
Abstract:
In this work, we aim to effectively characterize the performance of cooperative integrated sensing and communication (ISAC) networks and to reveal how performance metrics relate to network parameters. To this end, we introduce a generalized stochastic geometry framework to model the cooperative ISAC networks, which approximates the spatial randomness of the network deployment. Based on this framew…
▽ More
In this work, we aim to effectively characterize the performance of cooperative integrated sensing and communication (ISAC) networks and to reveal how performance metrics relate to network parameters. To this end, we introduce a generalized stochastic geometry framework to model the cooperative ISAC networks, which approximates the spatial randomness of the network deployment. Based on this framework, we derive analytical expressions for key performance metrics in both communication and sensing domains, with a particular focus on communication coverage probability and radar information rate. The analytical expressions derived explicitly highlight how performance metrics depend on network parameters, thereby offering valuable insights into the deployment and design of cooperative ISAC networks. In the end, we validate the theoretical performance analysis through Monte Carlo simulation results. Our results demonstrate that increasing the number of cooperative base stations (BSs) significantly improves both metrics, while increasing the BS deployment density has a limited impact on communication coverage probability but substantially enhances the radar information rate. Additionally, increasing the number of transmit antennas is effective when the total number of transmit antennas is relatively small. The incremental performance gain reduces with the increase of the number of transmit antennas, suggesting that indiscriminately increasing antennas is not an efficient strategy to improve the performance of the system in cooperative ISAC networks.
△ Less
Submitted 13 May, 2025; v1 submitted 13 May, 2025;
originally announced May 2025.
-
Vision-to-Music Generation: A Survey
Authors:
Zhaokai Wang,
Chenxi Bao,
Le Zhuo,
Jingrui Han,
Yang Yue,
Yihong Tang,
Victor Shea-Jay Huang,
Yue Liao
Abstract:
Vision-to-music Generation, including video-to-music and image-to-music tasks, is a significant branch of multimodal artificial intelligence demonstrating vast application prospects in fields such as film scoring, short video creation, and dance music synthesis. However, compared to the rapid development of modalities like text and images, research in vision-to-music is still in its preliminary st…
▽ More
Vision-to-music Generation, including video-to-music and image-to-music tasks, is a significant branch of multimodal artificial intelligence demonstrating vast application prospects in fields such as film scoring, short video creation, and dance music synthesis. However, compared to the rapid development of modalities like text and images, research in vision-to-music is still in its preliminary stage due to its complex internal structure and the difficulty of modeling dynamic relationships with video. Existing surveys focus on general music generation without comprehensive discussion on vision-to-music. In this paper, we systematically review the research progress in the field of vision-to-music generation. We first analyze the technical characteristics and core challenges for three input types: general videos, human movement videos, and images, as well as two output types of symbolic music and audio music. We then summarize the existing methodologies on vision-to-music generation from the architecture perspective. A detailed review of common datasets and evaluation metrics is provided. Finally, we discuss current challenges and promising directions for future research. We hope our survey can inspire further innovation in vision-to-music generation and the broader field of multimodal generation in academic research and industrial applications. To follow latest works and foster further innovation in this field, we are continuously maintaining a GitHub repository at https://github.com/wzk1015/Awesome-Vision-to-Music-Generation.
△ Less
Submitted 27 March, 2025;
originally announced March 2025.
-
Channel Estimation for RIS-Aided MU-MIMO mmWave Systems with Practical Hybrid Architecture
Authors:
Liuchang Zhuo,
Cunhua Pan,
Hong Ren,
Ruisong Weng,
Shi Jin,
A. Lee Swindlehurst,
Jiangzhou Wang
Abstract:
This paper proposes a correlation-based three-stage channel estimation strategy with low pilot overhead for reconfigurable intelligent surface (RIS)-aided millimeter wave (mmWave) multi-user (MU) MIMO systems, in which both users and base station (BS) are equipped with a hybrid RF architecture. In Stage I, all users jointly transmit pilots and recover the uncompressed received signals to estimate…
▽ More
This paper proposes a correlation-based three-stage channel estimation strategy with low pilot overhead for reconfigurable intelligent surface (RIS)-aided millimeter wave (mmWave) multi-user (MU) MIMO systems, in which both users and base station (BS) are equipped with a hybrid RF architecture. In Stage I, all users jointly transmit pilots and recover the uncompressed received signals to estimate the angle of arrival (AoA) at the BS using the discrete Fourier transform (DFT). Based on the observation that the overall cascaded MIMO channel can be decomposed into multiple sub-channels, the cascaded channel for a typical user is estimated in Stage II. Specifically, using the invariance of angles and the linear correlation of gains related to different cascaded subchannels, we use compressive sensing (CS), least squares (LS), and a one-dimensional search to estimate the Angles of Departure (AoDs), based on which the overall cascaded channel is obtained. In Stage III, the remaining users independently transmit pilots to estimate their individual cascaded channel with the same approach as in Stage II, which exploits the equivalent common RIS-BS channel obtained in Stage II to reduce the pilot overhead. In addition, the hybrid combining matrix and the RIS phase shift matrix are designed to reduce the noise power, thereby further improving the estimation performance. Simulation results demonstrate that the proposed algorithm can achieve high estimation accuracy especially when the number of antennas at the users is small, and reduce pilot overhead by more than five times compared with the existing benchmark approach.
△ Less
Submitted 8 February, 2025;
originally announced February 2025.
-
Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation
Authors:
Baisen Wang,
Le Zhuo,
Zhaokai Wang,
Chenxi Bao,
Wu Chengjing,
Xuecheng Nie,
Jiao Dai,
Jizhong Han,
Yue Liao,
Si Liu
Abstract:
Multimodal music generation aims to produce music from diverse input modalities, including text, videos, and images. Existing methods use a common embedding space for multimodal fusion. Despite their effectiveness in other modalities, their application in multimodal music generation faces challenges of data scarcity, weak cross-modal alignment, and limited controllability. This paper addresses the…
▽ More
Multimodal music generation aims to produce music from diverse input modalities, including text, videos, and images. Existing methods use a common embedding space for multimodal fusion. Despite their effectiveness in other modalities, their application in multimodal music generation faces challenges of data scarcity, weak cross-modal alignment, and limited controllability. This paper addresses these issues by using explicit bridges of text and music for multimodal alignment. We introduce a novel method named Visuals Music Bridge (VMB). Specifically, a Multimodal Music Description Model converts visual inputs into detailed textual descriptions to provide the text bridge; a Dual-track Music Retrieval module that combines broad and targeted retrieval strategies to provide the music bridge and enable user control. Finally, we design an Explicitly Conditioned Music Generation framework to generate music based on the two bridges. We conduct experiments on video-to-music, image-to-music, text-to-music, and controllable music generation tasks, along with experiments on controllability. The results demonstrate that VMB significantly enhances music quality, modality, and customization alignment compared to previous methods. VMB sets a new standard for interpretable and expressive multimodal music generation with applications in various multimedia fields. Demos and code are available at https://github.com/wbs2788/VMB.
△ Less
Submitted 12 December, 2024;
originally announced December 2024.
-
DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
Authors:
Qiaosong Qi,
Le Zhuo,
Aixi Zhang,
Yue Liao,
Fei Fang,
Si Liu,
Shuicheng Yan
Abstract:
When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we…
▽ More
When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. To bridge the gap between music and motion for conditional generation, DiffDance employs a pretrained audio representation learning model to extract music embeddings and further align its embedding space to motion via contrastive loss. During training our cascaded diffusion model, we also incorporate multiple geometric losses to constrain the model outputs to be physically plausible and add a dynamic loss weight that adaptively changes over diffusion timesteps to facilitate sample diversity. Through comprehensive experiments performed on the benchmark dataset AIST++, we demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music. These results are comparable to those achieved by state-of-the-art autoregressive methods.
△ Less
Submitted 5 August, 2023;
originally announced August 2023.
-
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
Authors:
Le Zhuo,
Ruibin Yuan,
Jiahao Pan,
Yinghao Ma,
Yizhi LI,
Ge Zhang,
Si Liu,
Roger Dannenberg,
Jie Fu,
Chenghua Lin,
Emmanouil Benetos,
Wei Xue,
Yike Guo
Abstract:
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets, even in challenging genres such as rock and metal. Our novel, training-free approach utilizes Whisper, a weakly supervised robust speech recognition model, and GPT-4, today's most performant chat-based large language mo…
▽ More
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets, even in challenging genres such as rock and metal. Our novel, training-free approach utilizes Whisper, a weakly supervised robust speech recognition model, and GPT-4, today's most performant chat-based large language model. In the proposed method, Whisper functions as the "ear" by transcribing the audio, while GPT-4 serves as the "brain," acting as an annotator with a strong performance for contextualized output selection and correction. Our experiments show that LyricWhiz significantly reduces Word Error Rate compared to existing methods in English and can effectively transcribe lyrics across multiple languages. Furthermore, we use LyricWhiz to create the first publicly available, large-scale, multilingual lyrics transcription dataset with a CC-BY-NC-SA copyright license, based on MTG-Jamendo, and offer a human-annotated subset for noise level estimation and evaluation. We anticipate that our proposed method and dataset will advance the development of multilingual lyrics transcription, a challenging and emerging task.
△ Less
Submitted 25 July, 2024; v1 submitted 29 June, 2023;
originally announced June 2023.
-
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
Authors:
Ruibin Yuan,
Yinghao Ma,
Yizhi Li,
Ge Zhang,
Xingran Chen,
Hanzhi Yin,
Le Zhuo,
Yiqi Liu,
Jiawen Huang,
Zeyue Tian,
Binyue Deng,
Ningzhi Wang,
Chenghua Lin,
Emmanouil Benetos,
Anton Ragni,
Norbert Gyenge,
Roger Dannenberg,
Wenhu Chen,
Gus Xia,
Wei Xue,
Si Liu,
Shi Wang,
Ruibo Liu,
Yike Guo,
Jie Fu
Abstract:
In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly in music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark. To address this issue…
▽ More
In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly in music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark. To address this issue, we introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines. Besides, MARBLE offers an easy-to-use, extendable, and reproducible suite for the community, with a clear statement on copyright issues on datasets. Results suggest recently proposed large-scale pre-trained musical language models perform the best in most tasks, with room for further improvement. The leaderboard and toolkit repository are published at https://marble-bm.shef.ac.uk to promote future music AI research.
△ Less
Submitted 23 November, 2023; v1 submitted 18 June, 2023;
originally announced June 2023.
-
Video Background Music Generation: Dataset, Method and Evaluation
Authors:
Le Zhuo,
Zhaokai Wang,
Baisen Wang,
Yue Liao,
Chenxi Bao,
Stanley Peng,
Songhao Han,
Aixi Zhang,
Fei Fang,
Si Liu
Abstract:
Music is essential when editing videos, but selecting music manually is difficult and time-consuming. Thus, we seek to automatically generate background music tracks given video input. This is a challenging task since it requires music-video datasets, efficient architectures for video-to-music generation, and reasonable metrics, none of which currently exist. To close this gap, we introduce a comp…
▽ More
Music is essential when editing videos, but selecting music manually is difficult and time-consuming. Thus, we seek to automatically generate background music tracks given video input. This is a challenging task since it requires music-video datasets, efficient architectures for video-to-music generation, and reasonable metrics, none of which currently exist. To close this gap, we introduce a complete recipe including dataset, benchmark model, and evaluation metric for video background music generation. We present SymMV, a video and symbolic music dataset with various musical annotations. To the best of our knowledge, it is the first video-music dataset with rich musical annotations. We also propose a benchmark video background music generation framework named V-MusProd, which utilizes music priors of chords, melody, and accompaniment along with video-music relations of semantic, color, and motion features. To address the lack of objective metrics for video-music correspondence, we design a retrieval-based metric VMCP built upon a powerful video-music representation learning model. Experiments show that with our dataset, V-MusProd outperforms the state-of-the-art method in both music quality and correspondence with videos. We believe our dataset, benchmark model, and evaluation metric will boost the development of video background music generation. Our dataset and code are available at https://github.com/zhuole1025/SymMV.
△ Less
Submitted 4 August, 2023; v1 submitted 21 November, 2022;
originally announced November 2022.
-
Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis
Authors:
Long Zhuo,
Guangcong Wang,
Shikai Li,
Wayne Wu,
Ziwei Liu
Abstract:
Video-to-Video synthesis (Vid2Vid) has achieved remarkable results in generating a photo-realistic video from a sequence of semantic maps. However, this pipeline suffers from high computational cost and long inference latency, which largely depends on two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have…
▽ More
Video-to-Video synthesis (Vid2Vid) has achieved remarkable results in generating a photo-realistic video from a sequence of semantic maps. However, this pipeline suffers from high computational cost and long inference latency, which largely depends on two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly compressed via more efficient network architectures. Nevertheless, existing methods mainly focus on slimming network architectures and ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal compression framework, \textbf{Fast-Vid2Vid}, which focuses on data aspects of generative models. It makes the first attempt at time dimension to reduce computational resources and accelerate inference. Specifically, we compress the input data stream spatially and reduce the temporal redundancy. After the proposed spatial-temporal knowledge distillation, our model can synthesize key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid interpolates intermediate frames by motion compensation with slight latency. On standard benchmarks, Fast-Vid2Vid achieves around real-time performance as 20 FPS and saves around 8x computational cost on a single V100 GPU.
△ Less
Submitted 11 July, 2022;
originally announced July 2022.
-
Blur-Attention: A boosting mechanism for non-uniform blurred image restoration
Authors:
Xiaoguang Li,
Feifan Yang,
Kin Man Lam,
Li Zhuo,
Jiafeng Li
Abstract:
Dynamic scene deblurring is a challenging problem in computer vision. It is difficult to accurately estimate the spatially varying blur kernel by traditional methods. Data-driven-based methods usually employ kernel-free end-to-end mapping schemes, which are apt to overlook the kernel estimation. To address this issue, we propose a blur-attention module to dynamically capture the spatially varying…
▽ More
Dynamic scene deblurring is a challenging problem in computer vision. It is difficult to accurately estimate the spatially varying blur kernel by traditional methods. Data-driven-based methods usually employ kernel-free end-to-end mapping schemes, which are apt to overlook the kernel estimation. To address this issue, we propose a blur-attention module to dynamically capture the spatially varying features of non-uniform blurred images. The module consists of a DenseBlock unit and a spatial attention unit with multi-pooling feature fusion, which can effectively extract complex spatially varying blur features. We design a multi-level residual connection structure to connect multiple blur-attention modules to form a blur-attention network. By introducing the blur-attention network into a conditional generation adversarial framework, we propose an end-to-end blind motion deblurring method, namely Blur-Attention-GAN (BAG), for a single image. Our method can adaptively select the weights of the extracted features according to the spatially varying blur features, and dynamically restore the images. Experimental results show that the deblurring capability of our method achieved outstanding objective performance in terms of PSNR, SSIM, and subjective visual quality. Furthermore, by visualizing the features extracted by the blur-attention module, comprehensive discussions are provided on its effectiveness.
△ Less
Submitted 19 August, 2020;
originally announced August 2020.
-
A lateral semicircular canal segmentation based geometric calibration for human temporal bone CT Image
Authors:
Xiaoguang Li,
Peng Fu,
Hongxia Yin,
ZhenChang Wang,
Li Zhuo,
Hui Zhang
Abstract:
Computed Tomography (CT) of the temporal bone has become an important method for diagnosing ear diseases. Due to the different posture of the subject and the settings of CT scanners, the CT image of the human temporal bone should be geometrically calibrated to ensure the symmetry of the bilateral anatomical structure. Manual calibration is a time-consuming task for radiologists and an important pr…
▽ More
Computed Tomography (CT) of the temporal bone has become an important method for diagnosing ear diseases. Due to the different posture of the subject and the settings of CT scanners, the CT image of the human temporal bone should be geometrically calibrated to ensure the symmetry of the bilateral anatomical structure. Manual calibration is a time-consuming task for radiologists and an important pre-processing step for further computer-aided CT analysis. We propose an automatic calibration algorithm for temporal bone CT images. The lateral semicircular canals (LSCs) are segmented as anchors at first. Then, we define a standard 3D coordinate system. The key step is the LSC segmentation. We design a novel 3D LSC segmentation encoder-decoder network, which introduces a 3D dilated convolution and a multi-pooling scheme for feature fusion in the encoding stage. The experimental results show that our LSC segmentation network achieved a higher segmentation accuracy. Our proposed method can help to perform calibration of temporal bone CT images efficiently.
△ Less
Submitted 28 June, 2020;
originally announced June 2020.
-
Hybrid Precoder and Combiner for Imperfect Beam Alignment in mmWave MIMO Systems
Authors:
Chandan Pradhan,
Ang Li,
Li Zhuo,
Yonghui Li,
Branka Vucetic
Abstract:
In this letter, we aim to design a robust hybrid precoder and combiner against beam misalignment in millimeter-wave (mmWave) communication systems. We consider the inclusion of the `error statistics' into the precoder and combiner design, where the array response that incorporates the distribution of the misalignment error is first derived. An iterative algorithm is then proposed to design the rob…
▽ More
In this letter, we aim to design a robust hybrid precoder and combiner against beam misalignment in millimeter-wave (mmWave) communication systems. We consider the inclusion of the `error statistics' into the precoder and combiner design, where the array response that incorporates the distribution of the misalignment error is first derived. An iterative algorithm is then proposed to design the robust hybrid precoder and combiner to maximize the array gain in the presence of beam misalignment. To further enhance the spectral efficiency, a second-stage digital precoder and combiner are included to mitigate the inter-stream interference. Numerical results show that the proposed robust hybrid precoder and combiner design can effectively alleviate the performance degradation incurred by beam misalignment.
△ Less
Submitted 21 March, 2019;
originally announced March 2019.