Showing 1–31 of 31 results for author: Asif, M

RIS-based Physical Layer Security for Integrated Sensing and Communication: A Comprehensive Survey

Authors: Yongxiao Li, Feroz Khan, Manzoor Ahmed, Aized Amin Soofi, Wali Ullah Khan, Chandan Kumar Sheemar, Muhammad Asif, Zhu Han

Abstract: Integrated Sensing and Communication (ISAC) is a crucial component of future wireless networks, enabling seamless integration of Communication and Sensing (C\&S) functionalities. However, ensuring security in ISAC systems remains a significant challenge, as both C\&S data are susceptible to adversarial threats. Physical Layer Security (PLS) has emerged as a key framework for mitigating these risks… ▽ More Integrated Sensing and Communication (ISAC) is a crucial component of future wireless networks, enabling seamless integration of Communication and Sensing (C\&S) functionalities. However, ensuring security in ISAC systems remains a significant challenge, as both C\&S data are susceptible to adversarial threats. Physical Layer Security (PLS) has emerged as a key framework for mitigating these risks at the transmission level. Reconfigurable Intelligent Surfaces (RIS) further enhance PLS by dynamically shaping the radio environment to improve both secrecy along with C\&S performance. This survey begins with an overview of RIS, PLS, and ISAC fundamentals, establishing a foundation for understanding their integration. The state-of-the-art RIS-assisted PLS approaches in ISAC systems are then categorized into passive RIS and Active RIS (ARIS) paradigms. Passive RIS-based techniques focus on optimizing system throughput, covert communication, and Secrecy Rates (SRs), alongside improving sensing Signal-to-Noise Ratio (SNR) and Weighted Sum Rate (WSR) under various constraints. ARIS-based strategies extend these capabilities by actively optimizing beamforming to enhance secrecy and covert rates while ensuring robust sensing under communication and security constraints. By reviewing both passive and ARIS-based security frameworks, this survey highlights the transformative role of RIS in strengthening ISAC security. Furthermore, it explores key optimization methodologies, technical challenges, and future research directions for integrating RIS with PLS to ensure secure and efficient ISAC in next-generation 6G wireless networks. △ Less

Submitted 22 March, 2025; originally announced March 2025.

RENO: Real-Time Neural Compression for 3D LiDAR Point Clouds

Authors: Kang You, Tong Chen, Dandan Ding, M. Salman Asif, Zhan Ma

Abstract: Despite the substantial advancements demonstrated by learning-based neural models in the LiDAR Point Cloud Compression (LPCC) task, realizing real-time compression - an indispensable criterion for numerous industrial applications - remains a formidable challenge. This paper proposes RENO, the first real-time neural codec for 3D LiDAR point clouds, achieving superior performance with a lightweight… ▽ More Despite the substantial advancements demonstrated by learning-based neural models in the LiDAR Point Cloud Compression (LPCC) task, realizing real-time compression - an indispensable criterion for numerous industrial applications - remains a formidable challenge. This paper proposes RENO, the first real-time neural codec for 3D LiDAR point clouds, achieving superior performance with a lightweight model. RENO skips the octree construction and directly builds upon the multiscale sparse tensor representation. Instead of the multi-stage inferring, RENO devises sparse occupancy codes, which exploit cross-scale correlation and derive voxels' occupancy in a one-shot manner, greatly saving processing time. Experimental results demonstrate that the proposed RENO achieves real-time coding speed, 10 fps at 14-bit depth on a desktop platform (e.g., one RTX 3090 GPU) for both encoding and decoding processes, while providing 12.25% and 48.34% bit-rate savings compared to G-PCCv23 and Draco, respectively, at a similar quality. RENO model size is merely 1MB, making it attractive for practical applications. The source code is available at https://github.com/NJUVISION/RENO. △ Less

Submitted 16 March, 2025; originally announced March 2025.

Gaussian is All You Need: A Unified Framework for Solving Inverse Problems via Diffusion Posterior Sampling

Authors: Nebiyou Yismaw, Ulugbek S. Kamilov, M. Salman Asif

Abstract: Diffusion models can generate a variety of high-quality images by modeling complex data distributions. Trained diffusion models can also be very effective image priors for solving inverse problems. Most of the existing diffusion-based methods integrate data consistency steps by approximating the likelihood function within the diffusion reverse sampling process. In this paper, we show that the exis… ▽ More Diffusion models can generate a variety of high-quality images by modeling complex data distributions. Trained diffusion models can also be very effective image priors for solving inverse problems. Most of the existing diffusion-based methods integrate data consistency steps by approximating the likelihood function within the diffusion reverse sampling process. In this paper, we show that the existing approximations are either insufficient or computationally inefficient. To address these issues, we propose a unified likelihood approximation method that incorporates a covariance correction term to enhance the performance and avoids propagating gradients through the diffusion model. The correction term, when integrated into the reverse diffusion sampling process, achieves better convergence towards the true data posterior for selected distributions and improves performance on real-world natural image datasets. Furthermore, we present an efficient way to factorize and invert the covariance matrix of the likelihood function for several inverse problems. Our comprehensive experiments demonstrate the effectiveness of our method over several existing approaches. Code available at https://github.com/CSIPlab/CoDPS. △ Less

Submitted 6 June, 2025; v1 submitted 13 September, 2024; originally announced September 2024.

Integration of Beyond Diagonal RIS and UAVs in 6G NTNs: Enhancing Aerial Connectivity

Authors: Wali Ullah Khan, Eva Lagunas, Asad Mahmood, Muhammad Asif, Manzoor Ahmed, Symeon Chatzinotas

Abstract: The reconfigurable intelligent surface (RIS) technology shows great potential in sixth-generation (6G) terrestrial and non-terrestrial networks (NTNs) since it can effectively change wireless settings to improve connectivity. Extensive research has been conducted on traditional RIS systems with diagonal phase response matrices. The straightforward RIS architecture, while cost-effective, has restri… ▽ More The reconfigurable intelligent surface (RIS) technology shows great potential in sixth-generation (6G) terrestrial and non-terrestrial networks (NTNs) since it can effectively change wireless settings to improve connectivity. Extensive research has been conducted on traditional RIS systems with diagonal phase response matrices. The straightforward RIS architecture, while cost-effective, has restricted capabilities in manipulating the wireless channels. The beyond diagonal reconfigurable intelligent surface (BD-RIS) greatly improves control over the wireless environment by utilizing interconnected phase response elements. This work proposes the integration of unmanned aerial vehicle (UAV) communications and BD-RIS in 6G NTNs, which has the potential to further enhance wireless coverage and spectral efficiency. We begin with the preliminaries of UAV communications and then discuss the fundamentals of BD-RIS technology. Subsequently, we discuss the potential of BD-RIS and UAV communications integration. We then proposed a case study based on UAV-mounted transmissive BD-RIS communication. Finally, we highlight future research directions and conclude this work. △ Less

Submitted 22 February, 2025; v1 submitted 9 September, 2024; originally announced September 2024.

Comments: 7,4

Overcoming Distribution Shifts in Plug-and-Play Methods with Test-Time Training

Authors: Edward P. Chandler, Shirin Shoushtari, Jiaming Liu, M. Salman Asif, Ulugbek S. Kamilov

Abstract: Plug-and-Play Priors (PnP) is a well-known class of methods for solving inverse problems in computational imaging. PnP methods combine physical forward models with learned prior models specified as image denoisers. A common issue with the learned models is that of a performance drop when there is a distribution shift between the training and testing data. Test-time training (TTT) was recently prop… ▽ More Plug-and-Play Priors (PnP) is a well-known class of methods for solving inverse problems in computational imaging. PnP methods combine physical forward models with learned prior models specified as image denoisers. A common issue with the learned models is that of a performance drop when there is a distribution shift between the training and testing data. Test-time training (TTT) was recently proposed as a general strategy for improving the performance of learned models when training and testing data come from different distributions. In this paper, we propose PnP-TTT as a new method for overcoming distribution shifts in PnP. PnP-TTT uses deep equilibrium learning (DEQ) for optimizing a self-supervised loss at the fixed points of PnP iterations. PnP-TTT can be directly applied on a single test sample to improve the generalization of PnP. We show through simulations that given a sufficient number of measurements, PnP-TTT enables the use of image priors trained on natural images for image reconstruction in magnetic resonance imaging (MRI). △ Less

Submitted 15 March, 2024; originally announced March 2024.

Journal ref: 2023 IEEE 9th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2023, pg. 186-190

Domain Expansion via Network Adaptation for Solving Inverse Problems

Authors: Nebiyou Yismaw, Ulugbek S. Kamilov, M. Salman Asif

Abstract: Deep learning-based methods deliver state-of-the-art performance for solving inverse problems that arise in computational imaging. These methods can be broadly divided into two groups: (1) learn a network to map measurements to the signal estimate, which is known to be fragile; (2) learn a prior for the signal to use in an optimization-based recovery. Despite the impressive results from the latter… ▽ More Deep learning-based methods deliver state-of-the-art performance for solving inverse problems that arise in computational imaging. These methods can be broadly divided into two groups: (1) learn a network to map measurements to the signal estimate, which is known to be fragile; (2) learn a prior for the signal to use in an optimization-based recovery. Despite the impressive results from the latter approach, many of these methods also lack robustness to shifts in data distribution, measurements, and noise levels. Such domain shifts result in a performance gap and in some cases introduce undesired artifacts in the estimated signal. In this paper, we explore the qualitative and quantitative effects of various domain shifts and propose a flexible and parameter efficient framework that adapt pretrained networks to such shifts. We demonstrate the effectiveness of our method for a number of natural image, MRI, and CT reconstructions tasks under domain, measurement model, and noise-level shifts. Our experiments demonstrate that our method provides significantly better performance and parameter efficiency compared to existing domain adaptation techniques. △ Less

Submitted 9 October, 2023; originally announced October 2023.

Inter Subject Emotion Recognition Using Spatio-Temporal Features From EEG Signal

Authors: Mohammad Asif, Diya Srivastava, Aditya Gupta, Uma Shanker Tiwary

Abstract: Inter-subject or subject-independent emotion recognition has been a challenging task in affective computing. This work is about an easy-to-implement emotion recognition model that classifies emotions from EEG signals subject independently. It is based on the famous EEGNet architecture, which is used in EEG-related BCIs. We used the Dataset on Emotion using Naturalistic Stimuli (DENS) dataset. The… ▽ More Inter-subject or subject-independent emotion recognition has been a challenging task in affective computing. This work is about an easy-to-implement emotion recognition model that classifies emotions from EEG signals subject independently. It is based on the famous EEGNet architecture, which is used in EEG-related BCIs. We used the Dataset on Emotion using Naturalistic Stimuli (DENS) dataset. The dataset contains the Emotional Events -- the precise information of the emotion timings that participants felt. The model is a combination of regular, depthwise and separable convolution layers of CNN to classify the emotions. The model has the capacity to learn the spatial features of the EEG channels and the temporal features of the EEG signals variability with time. The model is evaluated for the valence space ratings. The model achieved an accuracy of 73.04%. △ Less

Submitted 27 May, 2023; originally announced May 2023.

Report number: 2023 27th International Computer Science and Engineering Conference (ICSEC)

PIQI: Perceptual Image Quality Index based on Ensemble of Gaussian Process Regression

Authors: Nisar Ahmed, Hafiz Muhammad Shahzad Asif, Hassan Khalid

Abstract: Digital images contain a lot of redundancies, therefore, compression techniques are applied to reduce the image size without loss of reasonable image quality. Same become more prominent in the case of videos which contains image sequences and higher compression ratios are achieved in low throughput networks. Assessment of quality of images in such scenarios has become of particular interest. Subje… ▽ More Digital images contain a lot of redundancies, therefore, compression techniques are applied to reduce the image size without loss of reasonable image quality. Same become more prominent in the case of videos which contains image sequences and higher compression ratios are achieved in low throughput networks. Assessment of quality of images in such scenarios has become of particular interest. Subjective evaluation in most of the scenarios is infeasible so objective evaluation is preferred. Among the three objective quality measures, full-reference and reduced-reference methods require an original image in some form to calculate the image quality which is unfeasible in scenarios such as broadcasting, acquisition or enhancement. Therefore, a no-reference Perceptual Image Quality Index (PIQI) is proposed in this paper to assess the quality of digital images which calculates luminance and gradient statistics along with mean subtracted contrast normalized products in multiple scales and color spaces. These extracted features are provided to a stacked ensemble of Gaussian Process Regression (GPR) to perform the perceptual quality evaluation. The performance of the PIQI is checked on six benchmark databases and compared with twelve state-of-the-art methods and competitive results are achieved. The comparison is made based on RMSE, Pearson and Spearman correlation coefficients between ground truth and predicted quality scores. The scores of 0.0552, 0.9802 and 0.9776 are achieved respectively for these metrics on CSIQ database. Two cross-dataset evaluation experiments are performed to check the generalization of PIQI. △ Less

Submitted 16 May, 2023; originally announced May 2023.

Journal ref: AMultimed Tools Appl 80, 15677 to 15700 (2021)

Deep Ensembling for Perceptual Image Quality Assessment

Authors: Nisar Ahmed, H. M. Shahzad Asif, Abdul Rauf Bhatti, Atif Khan

Abstract: Blind image quality assessment is a challenging task particularly due to the unavailability of reference information. Training a deep neural network requires a large amount of training data which is not readily available for image quality. Transfer learning is usually opted to overcome this limitation and different deep architectures are used for this purpose as they learn features differently. Af… ▽ More Blind image quality assessment is a challenging task particularly due to the unavailability of reference information. Training a deep neural network requires a large amount of training data which is not readily available for image quality. Transfer learning is usually opted to overcome this limitation and different deep architectures are used for this purpose as they learn features differently. After extensive experiments, we have designed a deep architecture containing two CNN architectures as its sub-units. Moreover, a self-collected image database BIQ2021 is proposed with 12,000 images having natural distortions. The self-collected database is subjectively scored and is used for model training and validation. It is demonstrated that synthetic distortion databases cannot provide generalization beyond the distortion types used in the database and they are not ideal candidates for general-purpose image quality assessment. Moreover, a large-scale database of 18.75 million images with synthetic distortions is used to pretrain the model and then retrain it on benchmark databases for evaluation. Experiments are conducted on six benchmark databases three of which are synthetic distortion databases (LIVE, CSIQ and TID2013) and three are natural distortion databases (LIVE Challenge Database, CID2013 and KonIQ-10 k). The proposed approach has provided a Pearson correlation coefficient of 0.8992, 0.8472 and 0.9452 subsequently and Spearman correlation coefficient of 0.8863, 0.8408 and 0.9421. Moreover, the performance is demonstrated using perceptually weighted rank correlation to indicate the perceptual superiority of the proposed approach. Multiple experiments are conducted to validate the generalization performance of the proposed model by training on different subsets of the databases and validating on the test subset of BIQ2021 database. △ Less

Submitted 15 May, 2023; originally announced May 2023.

Journal ref: Soft Comput 26, 7601 to 7622 (2022)

On The Application Of Log Compression and Enhanced Denoising In Contrast Enhancement Of Digital Radiography Images

Authors: M. S. Asif, Mahesh Raveendranatha Panicker

Abstract: Digital radiography (DR) is becoming popular for the point of care imaging in the recent past. To reduce the radiation exposure, controlled radiation based on as low as reasonably achievable (ALARA) principle is employed and this results in low contrast images. To address this issue, post-processing algorithms such as the Multiscale Image Contrast Amplification (MUSICA) algorithm can be used to en… ▽ More Digital radiography (DR) is becoming popular for the point of care imaging in the recent past. To reduce the radiation exposure, controlled radiation based on as low as reasonably achievable (ALARA) principle is employed and this results in low contrast images. To address this issue, post-processing algorithms such as the Multiscale Image Contrast Amplification (MUSICA) algorithm can be used to enhance the contrast of DR images even with a low radiation dose. In this study, a modification of the MUSICA algorithm is investigated to determine the potential for further contrast improvement specifically for DR images. The conclusion is that combining log compression and its inverse at the appropriate stage with a multi-stage MUSICA and denoising is very promising. The proposed method resulted in an average of 66.5 % increase in the mean contrast-to-noise ratio (CNR) for the test images considered. △ Less

Submitted 18 April, 2023; v1 submitted 8 April, 2023; originally announced April 2023.

Comments: 4 pages, 4 figures

Fast Marching based Tissue Adaptive Delay Estimation for Aberration Corrected Delay and Sum Beamforming in Ultrasound Imaging

Authors: M. S. Asif, Gayathri Malamal, A. N. Madhavanunni, Vikram Melapudi, V Rahul, Abhijit Patil, Rajesh Langoju, Mahesh Raveendranatha Panicker

Abstract: Conventional ultrasound (US) imaging employs the delay and sum (DAS) receive beamforming with dynamic receive focus for image reconstruction due to its simplicity and robustness. However, the DAS beamforming follows a geometrical method of delay estimation with a spatially constant speed-of-sound (SoS) of 1540 m/s throughout the medium irrespective of the tissue in-homogeneity. This approximation… ▽ More Conventional ultrasound (US) imaging employs the delay and sum (DAS) receive beamforming with dynamic receive focus for image reconstruction due to its simplicity and robustness. However, the DAS beamforming follows a geometrical method of delay estimation with a spatially constant speed-of-sound (SoS) of 1540 m/s throughout the medium irrespective of the tissue in-homogeneity. This approximation leads to errors in delay estimations that accumulate with depth and degrades the resolution, contrast and overall accuracy of the US image. In this work, we propose a fast marching based DAS for focused transmissions which leverages the approximate SoS map to estimate the refraction corrected propagation delays for each pixel in the medium. The proposed approach is validated qualitatively and quantitatively for imaging depths of upto ~ 11 cm through simulations, where fat layer induced aberration is employed to alter the SoS in the medium. To the best of authors' knowledge, this is the first work considering the effect of SoS on image quality for deeper imaging. △ Less

Submitted 19 April, 2023; v1 submitted 7 April, 2023; originally announced April 2023.

Comments: 5 pages, 4 figures

Coded Illumination for 3D Lensless Imaging

Authors: Yucheng Zheng, M. Salman Asif

Abstract: Mask-based lensless cameras offer a novel design for imaging systems by replacing the lens in a conventional camera with a layer of coded mask. Each pixel of the lensless camera encodes the information of the entire 3D scene. Existing methods for 3D reconstruction from lensless measurements suffer from poor spatial and depth resolution. This is partially due to the system ill conditioning that ari… ▽ More Mask-based lensless cameras offer a novel design for imaging systems by replacing the lens in a conventional camera with a layer of coded mask. Each pixel of the lensless camera encodes the information of the entire 3D scene. Existing methods for 3D reconstruction from lensless measurements suffer from poor spatial and depth resolution. This is partially due to the system ill conditioning that arises because the point-spread functions (PSFs) from different depth planes are very similar. In this paper, we propose to capture multiple measurements of the scene under a sequence of coded illumination patterns to improve the 3D image reconstruction quality. In addition, we put the illumination source at a distance away from the camera. With such baseline distance between the lensless camera and illumination source, the camera observes a slice of the 3D volume, and the PSF of each depth plane becomes more resolvable from each other. We present simulation results along with experimental results with a camera prototype to demonstrate the effectiveness of our approach. △ Less

Submitted 22 December, 2022; originally announced December 2022.

Efficient Visual Computing with Camera RAW Snapshots

Authors: Zhihao Li, Ming Lu, Xu Zhang, Xin Feng, M. Salman Asif, Zhan Ma

Abstract: Conventional cameras capture image irradiance on a sensor and convert it to RGB images using an image signal processor (ISP). The images can then be used for photography or visual computing tasks in a variety of applications, such as public safety surveillance and autonomous driving. One can argue that since RAW images contain all the captured information, the conversion of RAW to RGB using an ISP… ▽ More Conventional cameras capture image irradiance on a sensor and convert it to RGB images using an image signal processor (ISP). The images can then be used for photography or visual computing tasks in a variety of applications, such as public safety surveillance and autonomous driving. One can argue that since RAW images contain all the captured information, the conversion of RAW to RGB using an ISP is not necessary for visual computing. In this paper, we propose a novel $ρ$-Vision framework to perform high-level semantic understanding and low-level compression using RAW images without the ISP subsystem used for decades. Considering the scarcity of available RAW image datasets, we first develop an unpaired CycleR2R network based on unsupervised CycleGAN to train modular unrolled ISP and inverse ISP (invISP) models using unpaired RAW and RGB images. We can then flexibly generate simulated RAW images (simRAW) using any existing RGB image dataset and finetune different models originally trained for the RGB domain to process real-world camera RAW images. We demonstrate object detection and image compression capabilities in RAW-domain using RAW-domain YOLOv3 and RAW image compressor (RIC) on snapshots from various cameras. Quantitative results reveal that RAW-domain task inference provides better detection accuracy and compression compared to RGB-domain processing. Furthermore, the proposed \r{ho}-Vision generalizes across various camera sensors and different task-specific models. Additional advantages of the proposed $ρ$-Vision that eliminates the ISP are the potential reductions in computations and processing times. △ Less

Submitted 25 January, 2024; v1 submitted 15 December, 2022; originally announced December 2022.

Comments: Accepted by T-PAMI 2024. Homepage: https://njuvision.github.io/rho-vision

Emotion Recognition With Temporarily Localized 'Emotional Events' in Naturalistic Context

Authors: Mohammad Asif, Sudhakar Mishra, Majithia Tejas Vinodbhai, Uma Shanker Tiwary

Abstract: Emotion recognition using EEG signals is an emerging area of research due to its broad applicability in BCI. Emotional feelings are hard to stimulate in the lab. Emotions do not last long, yet they need enough context to be perceived and felt. However, most EEG-related emotion databases either suffer from emotionally irrelevant details (due to prolonged duration stimulus) or have minimal context d… ▽ More Emotion recognition using EEG signals is an emerging area of research due to its broad applicability in BCI. Emotional feelings are hard to stimulate in the lab. Emotions do not last long, yet they need enough context to be perceived and felt. However, most EEG-related emotion databases either suffer from emotionally irrelevant details (due to prolonged duration stimulus) or have minimal context doubting the feeling of any emotion using the stimulus. We tried to reduce the impact of this trade-off by designing an experiment in which participants are free to report their emotional feelings simultaneously watching the emotional stimulus. We called these reported emotional feelings "Emotional Events" in our Dataset on Emotion with Naturalistic Stimuli (DENS). We used EEG signals to classify emotional events on different combinations of Valence(V) and Arousal(A) dimensions and compared the results with benchmark datasets of DEAP and SEED. STFT is used for feature extraction and used in the classification model consisting of CNN-LSTM hybrid layers. We achieved significantly higher accuracy with our data compared to DEEP and SEED data. We conclude that having precise information about emotional feelings improves the classification accuracy compared to long-duration EEG signals which might be contaminated by mind-wandering. △ Less

Submitted 25 October, 2022; originally announced November 2022.

Energy-Efficient Beamforming and Resource Optimization for AmBSC-Assisted Cooperative NOMA IoT Networks

Authors: Muhammad Asif, Asim Ihsan, Wali Ullah Khan, Ali Ranjha, Shengli Zhang, Sissi Xiaoxiao Wu

Abstract: In this manuscript, we present an energy-efficient alternating optimization framework based on the multi-antenna ambient backscatter communication (AmBSC) assisted cooperative non-orthogonal multiple access (NOMA) for next-generation (NG) internet-of-things (IoT) enabled communication networks. Specifically, the energy-efficiency maximization is achieved for the considered AmBSC-enabled multi-clus… ▽ More In this manuscript, we present an energy-efficient alternating optimization framework based on the multi-antenna ambient backscatter communication (AmBSC) assisted cooperative non-orthogonal multiple access (NOMA) for next-generation (NG) internet-of-things (IoT) enabled communication networks. Specifically, the energy-efficiency maximization is achieved for the considered AmBSC-enabled multi-cluster cooperative IoT NOMA system by optimizing the active-beamforming vector and power-allocation coefficients (PAC) of IoT NOMA users at the transmitter, as well as passive-beamforming vector at the multi-antenna assisted backscatter node. Usually, increasing the number of IoT NOMA users in each cluster results in inter-cluster interference (ICI) (among different clusters) and intra-cluster interference (among IoT NOMA users). To combat the impact of ICI, we exploit a zero-forcing (ZF) based active-beamforming, as well as an efficient clustering technique at the source node. Further, the effect of intra-cluster interference is mitigated by exploiting an efficient power-allocation policy that determines the PAC of IoT NOMA users under the quality-of-service (QoS), cooperation, SIC decoding, and power-budget constraints. Moreover, the considered non-convex passive-beamforming problem is transformed into a standard semi-definite programming (SDP) problem by exploiting the successive-convex approximation (SCA) approximation, as well as the difference of convex (DC) programming, where Rank-1 solution of passive-beamforming is obtained based on the penalty-based method. Furthermore, the numerical analysis of simulation results demonstrates that the proposed energy-efficiency maximization algorithm exhibits an efficient performance by achieving convergence within only a few iterations. △ Less

Submitted 11 September, 2022; originally announced September 2022.

Rate Splitting Multiple Access for Next Generation Cognitive Radio Enabled LEO Satellite Networks

Authors: ali Ullah Khan, Zain Ali, Eva Lagunas, Asad Mahmood, Muhammad Asif, Asim Ihsan, Symeon Chatzinotas, Björn Ottersten, Octavia A. Dobre

Abstract: This paper proposes a cognitive radio enabled LEO SatCom using RSMA radio access technique with the coexistence of GEO SatCom network. In particular, this work aims to maximize the sum rate of LEO SatCom by simultaneously optimizing the power budget over different beams, RSMA power allocation for users over each beam, and subcarrier user assignment while restricting the interference temperature to… ▽ More This paper proposes a cognitive radio enabled LEO SatCom using RSMA radio access technique with the coexistence of GEO SatCom network. In particular, this work aims to maximize the sum rate of LEO SatCom by simultaneously optimizing the power budget over different beams, RSMA power allocation for users over each beam, and subcarrier user assignment while restricting the interference temperature to GEO SatCom. The problem of sum rate maximization is formulated as non-convex, where the global optimal solution is challenging to obtain. Thus, an efficient solution can be obtained in three steps: first we employ a successive convex approximation technique to reduce the complexity and make the problem more tractable. Second, for any given resource block user assignment, we adopt KKT conditions to calculate the transmit power over different beams and RSMA power allocation of users over each beam. Third, using the allocated power, we design an efficient algorithm based on the greedy approach for resource block user assignment. Numerical results demonstrate the benefits of the proposed optimization scheme compared to the benchmark schemes. △ Less

Submitted 6 February, 2023; v1 submitted 7 August, 2022; originally announced August 2022.

Comments: 32,9. arXiv admin note: substantial text overlap with arXiv:2208.02924

Energy-Efficient Backscatter-Assisted Coded Cooperative-NOMA for B5G Wireless Communications

Authors: Muhammad Asif, Asim Ihsan, Wali Ullah Khan, Ali Ranjha, Shengli Zhang, Sissi Xiaoxiao Wu

Abstract: In this manuscript, we propose an alternating optimization framework to maximize the energy efficiency of a backscatter-enabled cooperative Non-orthogonal multiple access (NOMA) system by optimizing the transmit power of the source, power allocation coefficients (PAC), and power of the relay node under imperfect successive interference cancellation (SIC) decoding. A three-stage low-complexity ener… ▽ More In this manuscript, we propose an alternating optimization framework to maximize the energy efficiency of a backscatter-enabled cooperative Non-orthogonal multiple access (NOMA) system by optimizing the transmit power of the source, power allocation coefficients (PAC), and power of the relay node under imperfect successive interference cancellation (SIC) decoding. A three-stage low-complexity energy-efficient alternating optimization algorithm is introduced which optimizes the transmit power, PAC, and relay power by considering the quality of service (QoS), power budget, and cooperation constraints. Subsequently, a joint channel coding framework is introduced to enhance the performance of far user which has no direct communication link with the base station (BS) and has bad channel conditions. In the destination node, the far user data is jointly decoded using a Sum-product algorithm (SPA) based joint iterative decoder realized by jointly-designed Quasi-cyclic Low-density parity-check (QC-LDPC) codes. Simulation results evince that the proposed backscatter-enabled cooperative NOMA system outperforms its counterpart by providing an efficient performance in terms of energy efficiency. Also, proposed jointly-designed QC-LDPC codes provide an excellent bit-error-rate (BER) performance by jointly decoding the far user data for considered BSC cooperative NOMA system with only a few decoding iterations. △ Less

Submitted 20 August, 2022; v1 submitted 1 August, 2022; originally announced August 2022.

Comments: 30, 7

Cooperative Backscatter NOMA with Imperfect SIC: Towards Energy Efficient Sum Rate Maximization in Sustainable 6G Networks

Authors: Manzoor Ahmed, Zain Ali, Wali Ullah Khan, Omer Waqar, Muhammad Asif, Abd Ullah Khan, Muhammad Awais Javed, Fahd N. Al-Wesabi

Abstract: The combination of backscatter communication with non-orthogonal multiple access (NOMA) has the potential to support low-powered massive connections in upcoming sixth-generation (6G) wireless networks. More specifically, backscatter communication can harvest and use the existing RF signals in the atmosphere for communication, while NOMA provides communication to multiple wireless devices over the… ▽ More The combination of backscatter communication with non-orthogonal multiple access (NOMA) has the potential to support low-powered massive connections in upcoming sixth-generation (6G) wireless networks. More specifically, backscatter communication can harvest and use the existing RF signals in the atmosphere for communication, while NOMA provides communication to multiple wireless devices over the same frequency and time resources. This paper has proposed a new resource management framework for backscatter-aided cooperative NOMA communication in upcoming 6G networks. In particular, the proposed work has simultaneously optimized the base station's transmit power, relaying node, the reflection coefficient of the backscatter tag, and time allocation under imperfect successive interference cancellation to maximize the sum rate of the system. To obtain an efficient solution for the resource management framework, we have proposed a combination of the bisection method and dual theory, where the sub-gradient method is adopted to optimize the Lagrangian multipliers. Numerical results have shown that the proposed solution provides excellent performance. When the performance of the proposed technique is compared to a brute-forcing search technique that guarantees optimal solution however, is very time-consuming, it was seen that the gap in performance is actually 0\%. Hence, the proposed framework has provided performance equal to a cumbersome brute-force search technique while offering much less complexity. The works in the literature on cooperative NOMA considered equal time distribution for cooperation and direct communication. Our results showed that optimizing the time-division can increase the performance by more than 110\% for high transmission powers. △ Less

Submitted 7 July, 2022; originally announced July 2022.

Comments: 9, 7

LSTM-Based Distributed Conditional Generative Adversarial Network For Data-Driven 5G-Enabled Maritime UAV Communications

Authors: Iftikhar Rasheed, Muhammad Asif, Asim Ihsan, Wali Ullah Khan, Manzoor Ahmed, Khaled Rabie

Abstract: 5G enabled maritime unmanned aerial vehicle (UAV) communication is one of the important applications of 5G wireless network which requires minimum latency and higher reliability to support mission-critical applications. Therefore, lossless reliable communication with a high data rate is the key requirement in modern wireless communication systems. These all factors highly depend upon channel condi… ▽ More 5G enabled maritime unmanned aerial vehicle (UAV) communication is one of the important applications of 5G wireless network which requires minimum latency and higher reliability to support mission-critical applications. Therefore, lossless reliable communication with a high data rate is the key requirement in modern wireless communication systems. These all factors highly depend upon channel conditions. In this work, a channel model is proposed for air-to-surface link exploiting millimeter wave (mmWave) for 5G enabled maritime unmanned aerial vehicle (UAV) communication. Firstly, we will present the formulated channel estimation method which directly aims to adopt channel state information (CSI) of mmWave from the channel model inculcated by UAV operating within the Long Short Term Memory (LSTM)-Distributed Conditional generative adversarial network (DCGAN) i.e. (LSTM-DCGAN) for each beamforming direction. Secondly, to enhance the applications for the proposed trained channel model for the spatial domain, we have designed an LSTM-DCGAN based UAV network, where each one will learn mmWave CSI for all the distributions. Lastly, we have categorized the most favorable LSTM-DCGAN training method and emanated certain conditions for our UAV network to increase the channel model learning rate. Simulation results have shown that the proposed LSTM-DCGAN based network is vigorous to the error generated through local training. A detailed comparison has been done with the other available state-of-the-art CGAN network architectures i.e. stand-alone CGAN (without CSI sharing), Simple CGAN (with CSI sharing), multi-discriminator CGAN, federated learning CGAN and DCGAN. Simulation results have shown that the proposed LSTM-DCGAN structure demonstrates higher accuracy during the learning process and attained more data rate for downlink transmission as compared to the previous state of artworks. △ Less

Submitted 9 May, 2022; originally announced May 2022.

Comments: 14,7

Energy-Efficient IRS-Aided NOMA Beamforming for 6G Wireless Communications

Authors: Asim Ihsan, Wen Chen, Muhammad Asif, Wali Ullah Khan, Jun Li

Abstract: This manuscript presents an energy-efficient alternating optimization framework based on intelligent reflective surfaces (IRS) aided non-orthogonal multiple access beamforming (NOMA-BF) system for 6G wireless communications. Specifically, this work proposes a centralized IRS-enabled design for the NOMA-BF system to optimize the active beamforming and power allocation coefficient (PAC) of users at… ▽ More This manuscript presents an energy-efficient alternating optimization framework based on intelligent reflective surfaces (IRS) aided non-orthogonal multiple access beamforming (NOMA-BF) system for 6G wireless communications. Specifically, this work proposes a centralized IRS-enabled design for the NOMA-BF system to optimize the active beamforming and power allocation coefficient (PAC) of users at the transmitter in the first stage and passive beamforming at IRS in the 2nd stage to maximize the energy efficiency (EE) of the network. However, an increment in the number of supportable users with the NOMA-BF system will lead to NOMA user interference and inter-cluster interference (ICI). To mitigate the effect of ICI, first zero-forcing beamforming along with efficient user clustering algorithm is exploited and then NOMA user interference is tackled efficiently through a proposed iterative algorithm that computes PAC of NOMA user through simplified closed-form expression under the required system constraints. In the 2nd stage, the problem of passive beamforming is solved through a technique based on difference-of-convex (DC) programming and successive convex approximation (SCA). Simulation results demonstrate that the proposed alternating framework for energy-efficient IRS-assisted NOMA-BF system can achieve convergence within a few iterations and provide efficient performance in terms of EE of the system with low complexity. △ Less

Submitted 30 March, 2022; originally announced March 2022.

Comments: 12 pages, 8 figures. Submitted to IEEE

STRIDE-based Cyber Security Threat Modeling for IoT-enabled Precision Agriculture Systems

Authors: Md. Rashid Al Asif, Khondokar Fida Hasan, Md Zahidul Islam, Rahamatullah Khondoker

Abstract: The concept of traditional farming is changing rapidly with the introduction of smart technologies like the Internet of Things (IoT). Under the concept of smart agriculture, precision agriculture is gaining popularity to enable Decision Support System (DSS)-based farming management that utilizes widespread IoT sensors and wireless connectivity to enable automated detection and optimization of reso… ▽ More The concept of traditional farming is changing rapidly with the introduction of smart technologies like the Internet of Things (IoT). Under the concept of smart agriculture, precision agriculture is gaining popularity to enable Decision Support System (DSS)-based farming management that utilizes widespread IoT sensors and wireless connectivity to enable automated detection and optimization of resources. Undoubtedly the success of the system would be impacted on crop productivity, where failure would impact severely. Like many other cyber-physical systems, one of the growing challenges to avoid system adversity is to ensure the system's security, privacy, and trust. But what are the vulnerabilities, threats, and security issues we should consider while deploying precision agriculture? This paper has conducted a holistic threat modeling on component levels of precision agriculture's standard infrastructure using popular threat intelligence tools STRIDE to identify common security issues. Our modeling identifies a noticing of fifty-eight potential security threats to consider. This presentation systematically presented them and advised general mitigation suggestions to support cyber security in precision agriculture. △ Less

Submitted 30 January, 2022; v1 submitted 24 January, 2022; originally announced January 2022.

Non-Reference Quality Monitoring of Digital Images using Gradient Statistics and Feedforward Neural Networks

Authors: Nisar Ahmed, Hafiz Muhammad Shahzad Asif, Hassan Khalid

Abstract: Digital images contain a lot of redundancies, therefore, compressions are applied to reduce the image size without the loss of reasonable image quality. The same become more prominent in the case of videos that contains image sequences and higher compression ratios are achieved in low throughput networks. Assessment of the quality of images in such scenarios becomes of particular interest. Subject… ▽ More Digital images contain a lot of redundancies, therefore, compressions are applied to reduce the image size without the loss of reasonable image quality. The same become more prominent in the case of videos that contains image sequences and higher compression ratios are achieved in low throughput networks. Assessment of the quality of images in such scenarios becomes of particular interest. Subjective evaluation in most of the scenarios becomes infeasible so objective evaluation is preferred. Among the three objective quality measures, full-reference and reduced-reference methods require an original image in some form to calculate the quality score which is not feasible in scenarios such as broadcasting or IP video. Therefore, a non-reference quality metric is proposed to assess the quality of digital images which calculates luminance and multiscale gradient statistics along with mean subtracted contrast normalized products as features to train a Feedforward Neural Network with Scaled Conjugate Gradient. The trained network has provided good regression and R2 measures and further testing on LIVE Image Quality Assessment database release-2 has shown promising results. Pearson, Kendall, and Spearman's correlation are calculated between predicted and actual quality scores and their results are comparable to the state-of-the-art systems. Moreover, the proposed metric is computationally faster than its counterparts and can be used for the quality assessment of image sequences. △ Less

Submitted 27 December, 2021; originally announced December 2021.

Comments: Fifth International Conference on Aerospace Science & Engineering (ICASE 2017) (ICASE Proceedings, Page No. 300-305)

MSC Class: 94A08 ACM Class: I.4.5; I.5.4

Coded Illumination for Improved Lensless Imaging

Authors: Yucheng Zheng, M. Salman Asif

Abstract: Mask-based lensless cameras can be flat, thin, and light-weight, which makes them suitable for novel designs of computational imaging systems with large surface areas and arbitrary shapes. Despite recent progress in lensless cameras, the quality of images recovered from the lensless cameras is often poor due to the ill-conditioning of the underlying measurement system. In this paper, we propose to… ▽ More Mask-based lensless cameras can be flat, thin, and light-weight, which makes them suitable for novel designs of computational imaging systems with large surface areas and arbitrary shapes. Despite recent progress in lensless cameras, the quality of images recovered from the lensless cameras is often poor due to the ill-conditioning of the underlying measurement system. In this paper, we propose to use coded illumination to improve the quality of images reconstructed with lensless cameras. In our imaging model, the scene/object is illuminated by multiple coded illumination patterns as the lensless camera records sensor measurements. We designed and tested a number of illumination patterns and observed that shifting dots (and related orthogonal) patterns provide the best overall performance. We propose a fast and low-complexity recovery algorithm that exploits the separability and block-diagonal structure in our system. We present simulation results and hardware experiment results to demonstrate that our proposed method can significantly improve the reconstruction quality. △ Less

Submitted 9 January, 2023; v1 submitted 24 November, 2021; originally announced November 2021.

Comments: Supplementary material, codes, and data are available at https://github.com/CSIPlab/codedcam

Journal ref: IEEE Transactions on Computational Imaging, 2023

A Simple Framework for 3D Lensless Imaging with Programmable Masks

Authors: Yucheng Zheng, Yi Hua, Aswin C. Sankaranarayanan, M. Salman Asif

Abstract: Lensless cameras provide a framework to build thin imaging systems by replacing the lens in a conventional camera with an amplitude or phase mask near the sensor. Existing methods for lensless imaging can recover the depth and intensity of the scene, but they require solving computationally-expensive inverse problems. Furthermore, existing methods struggle to recover dense scenes with large depth… ▽ More Lensless cameras provide a framework to build thin imaging systems by replacing the lens in a conventional camera with an amplitude or phase mask near the sensor. Existing methods for lensless imaging can recover the depth and intensity of the scene, but they require solving computationally-expensive inverse problems. Furthermore, existing methods struggle to recover dense scenes with large depth variations. In this paper, we propose a lensless imaging system that captures a small number of measurements using different patterns on a programmable mask. In this context, we make three contributions. First, we present a fast recovery algorithm to recover textures on a fixed number of depth planes in the scene. Second, we consider the mask design problem, for programmable lensless cameras, and provide a design template for optimizing the mask patterns with the goal of improving depth estimation. Third, we use a refinement network as a post-processing step to identify and remove artifacts in the reconstruction. These modifications are evaluated extensively with experimental results on a lensless camera prototype to showcase the performance benefits of the optimized masks and recovery algorithms over the state of the art. △ Less

Submitted 18 August, 2021; originally announced August 2021.

Comments: Supplementary material available at https://github.com/CSIPlab/Programmable3Dcam.git

Journal ref: International Conference on Computer Vision (ICCV) 2021

Recovery Analysis for Plug-and-Play Priors using the Restricted Eigenvalue Condition

Authors: Jiaming Liu, M. Salman Asif, Brendt Wohlberg, Ulugbek S. Kamilov

Abstract: The plug-and-play priors (PnP) and regularization by denoising (RED) methods have become widely used for solving inverse problems by leveraging pre-trained deep denoisers as image priors. While the empirical imaging performance and the theoretical convergence properties of these algorithms have been widely investigated, their recovery properties have not previously been theoretically analyzed. We… ▽ More The plug-and-play priors (PnP) and regularization by denoising (RED) methods have become widely used for solving inverse problems by leveraging pre-trained deep denoisers as image priors. While the empirical imaging performance and the theoretical convergence properties of these algorithms have been widely investigated, their recovery properties have not previously been theoretically analyzed. We address this gap by showing how to establish theoretical recovery guarantees for PnP/RED by assuming that the solution of these methods lies near the fixed-points of a deep neural network. We also present numerical results comparing the recovery performance of PnP/RED in compressive sensing against that of recent compressive sensing algorithms based on generative models. Our numerical results suggest that PnP with a pre-trained artifact removal network provides significantly better results compared to the existing state-of-the-art methods. △ Less

Submitted 26 October, 2021; v1 submitted 7 June, 2021; originally announced June 2021.

Comments: 27 pages, 13 figures

Solving Phase Retrieval with a Learned Reference

Authors: Rakib Hyder, Zikui Cai, M. Salman Asif

Abstract: Fourier phase retrieval is a classical problem that deals with the recovery of an image from the amplitude measurements of its Fourier coefficients. Conventional methods solve this problem via iterative (alternating) minimization by leveraging some prior knowledge about the structure of the unknown image. The inherent ambiguities about shift and flip in the Fourier measurements make this problem e… ▽ More Fourier phase retrieval is a classical problem that deals with the recovery of an image from the amplitude measurements of its Fourier coefficients. Conventional methods solve this problem via iterative (alternating) minimization by leveraging some prior knowledge about the structure of the unknown image. The inherent ambiguities about shift and flip in the Fourier measurements make this problem especially difficult; and most of the existing methods use several random restarts with different permutations. In this paper, we assume that a known (learned) reference is added to the signal before capturing the Fourier amplitude measurements. Our method is inspired by the principle of adding a reference signal in holography. To recover the signal, we implement an iterative phase retrieval method as an unrolled network. Then we use back propagation to learn the reference that provides us the best reconstruction for a fixed number of phase retrieval iterations. We performed a number of simulations on a variety of datasets under different conditions and found that our proposed method for phase retrieval via unrolled network and learned reference provides near-perfect recovery at fixed (small) computational cost. We compared our method with standard Fourier phase retrieval methods and observed significant performance enhancement using the learned reference. △ Less

Submitted 29 July, 2020; originally announced July 2020.

Comments: Accepted to ECCV 2020. Code is available at https://github.com/CSIPlab/learnPR_reference

Learning Illumination Patterns for Coded Diffraction Phase Retrieval

Authors: Zikui Cai, Rakib Hyder, M. Salman Asif

Abstract: Signal recovery from nonlinear measurements involves solving an iterative optimization problem. In this paper, we present a framework to optimize the sensing parameters to improve the quality of the signal recovered by the given iterative method. In particular, we learn illumination patterns to recover signals from coded diffraction patterns using a fixed-cost alternating minimization-based phase… ▽ More Signal recovery from nonlinear measurements involves solving an iterative optimization problem. In this paper, we present a framework to optimize the sensing parameters to improve the quality of the signal recovered by the given iterative method. In particular, we learn illumination patterns to recover signals from coded diffraction patterns using a fixed-cost alternating minimization-based phase retrieval method. Coded diffraction phase retrieval is a physically realistic system in which the signal is first modulated by a sequence of codes before the sensor records its Fourier amplitude. We represent the phase retrieval method as an unrolled network with a fixed number of layers and minimize the recovery error by optimizing over the measurement parameters. Since the number of iterations/layers are fixed, the recovery incurs a fixed cost. We present extensive simulation results on a variety of datasets under different conditions and a comparison with existing methods. Our results demonstrate that the proposed method provides near-perfect reconstruction using patterns learned with a small number of training images. Our proposed method provides significant improvements over existing methods both in terms of accuracy and speed. △ Less

Submitted 7 June, 2020; originally announced June 2020.

Non-Adversarial Video Synthesis with Learned Priors

Authors: Abhishek Aich, Akash Gupta, Rameswar Panda, Rakib Hyder, M. Salman Asif, Amit K. Roy-Chowdhury

Abstract: Most of the existing works in video synthesis focus on generating videos using adversarial learning. Despite their success, these methods often require input reference frame or fail to generate diverse videos from the given data distribution, with little to no uniformity in the quality of videos that can be generated. Different from these methods, we focus on the problem of generating videos from… ▽ More Most of the existing works in video synthesis focus on generating videos using adversarial learning. Despite their success, these methods often require input reference frame or fail to generate diverse videos from the given data distribution, with little to no uniformity in the quality of videos that can be generated. Different from these methods, we focus on the problem of generating videos from latent noise vectors, without any reference input frames. To this end, we develop a novel approach that jointly optimizes the input latent space, the weights of a recurrent neural network and a generator through non-adversarial learning. Optimizing for the input latent space along with the network weights allows us to generate videos in a controlled environment, i.e., we can faithfully generate all videos the model has seen during the learning process as well as new unseen videos. Extensive experiments on three challenging and diverse datasets well demonstrate that our approach generates superior quality videos compared to the existing state-of-the-art methods. △ Less

Submitted 17 April, 2020; v1 submitted 20 March, 2020; originally announced March 2020.

Comments: Accepted to CVPR 2020

Implicit Higher-Order Moment Matching Technique for Model Reduction of Quadratic-bilinear Systems

Authors: Mian Muhammad Arsalan Asif, Mian Ilyas Ahmad, Peter Benner, Lihong Feng, Tatjana Stykel

Abstract: We propose a projection based multi-moment matching method for model order reduction of quadratic-bilinear systems. The goal is to construct a reduced system that ensures higher-order moment matching for the multivariate transfer functions appearing in the input-output representation of the nonlinear system. An existing technique achieves this for the first two multivariate transfer functions, in… ▽ More We propose a projection based multi-moment matching method for model order reduction of quadratic-bilinear systems. The goal is to construct a reduced system that ensures higher-order moment matching for the multivariate transfer functions appearing in the input-output representation of the nonlinear system. An existing technique achieves this for the first two multivariate transfer functions, in what is called the symmetric form of the multivariate transfer functions. We extend this framework to an equivalent and simplified form, the regular form, which allows us to show moment matching for the first three multivariate transfer functions. Numerical results for three benchmark examples of quadratic-bilinear systems show that the proposed framework exhibits better performance with reduced computational cost in comparison to existing techniques. △ Less

Submitted 13 November, 2019; originally announced November 2019.

Comments: 19 pages, 11 subfigures in 6 figures, Journal

MSC Class: 35G50

Joint Image and Depth Estimation with Mask-Based Lensless Cameras

Authors: Yucheng Zheng, M. Salman Asif

Abstract: Mask-based lensless cameras replace the lens of a conventional camera with a custom mask. These cameras can potentially be very thin and even flexible. Recently, it has been demonstrated that such mask-based cameras can recover light intensity and depth information of a scene. Existing depth recovery algorithms either assume that the scene consists of a small number of depth planes or solve a spar… ▽ More Mask-based lensless cameras replace the lens of a conventional camera with a custom mask. These cameras can potentially be very thin and even flexible. Recently, it has been demonstrated that such mask-based cameras can recover light intensity and depth information of a scene. Existing depth recovery algorithms either assume that the scene consists of a small number of depth planes or solve a sparse recovery problem over a large 3D volume. Both these approaches fail to recover the scenes with large depth variations. In this paper, we propose a new approach for depth estimation based on an alternating gradient descent algorithm that jointly estimates a continuous depth map and light distribution of the unknown scene from its lensless measurements. We present simulation results on image and depth reconstruction for a variety of 3D test scenes. A comparison between the proposed algorithm and other method shows that our algorithm is more robust for natural scenes with a large range of depths. We built a prototype lensless camera and present experimental results for reconstruction of intensity and depth maps of different real objects. △ Less

Submitted 19 June, 2020; v1 submitted 6 October, 2019; originally announced October 2019.

Comments: Added new experiments with camera prototype

A Dual Camera System for High Spatiotemporal Resolution Video Acquisition

Authors: Ming Cheng, Zhan Ma, M. Salman Asif, Yiling Xu, Haojie Liu, Wenbo Bao, Jun Sun

Abstract: This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end lea… ▽ More This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects (such as reference structure, camera parallax, exposure time, etc) of our system to fully understand its capability for potential applications. △ Less

Submitted 24 March, 2020; v1 submitted 28 September, 2019; originally announced September 2019.

Comments: To appear in IEEE Transactions on Pattern Analysis and Machine Intelligence