-
LLVD: LSTM-based Explicit Motion Modeling in Latent Space for Blind Video Denoising
Authors:
Loay Rashid,
Siddharth Roheda,
Amit Unde
Abstract:
Video restoration plays a pivotal role in revitalizing degraded video content by rectifying imperfections caused by various degradations introduced during capturing (sensor noise, motion blur, etc.), saving/sharing (compression, resizing, etc.) and editing. This paper introduces a novel algorithm designed for scenarios where noise is introduced during video capture, aiming to enhance the visual qu…
▽ More
Video restoration plays a pivotal role in revitalizing degraded video content by rectifying imperfections caused by various degradations introduced during capturing (sensor noise, motion blur, etc.), saving/sharing (compression, resizing, etc.) and editing. This paper introduces a novel algorithm designed for scenarios where noise is introduced during video capture, aiming to enhance the visual quality of videos by reducing unwanted noise artifacts. We propose the Latent space LSTM Video Denoiser (LLVD), an end-to-end blind denoising model. LLVD uniquely combines spatial and temporal feature extraction, employing Long Short Term Memory (LSTM) within the encoded feature domain. This integration of LSTM layers is crucial for maintaining continuity and minimizing flicker in the restored video. Moreover, processing frames in the encoded feature domain significantly reduces computations, resulting in a very lightweight architecture. LLVD's blind nature makes it versatile for real, in-the-wild denoising scenarios where prior information about noise characteristics is not available. Experiments reveal that LLVD demonstrates excellent performance for both synthetic and captured noise. Specifically, LLVD surpasses the current State-Of-The-Art (SOTA) in RAW denoising by 0.3dB, while also achieving a 59\% reduction in computational complexity.
△ Less
Submitted 10 January, 2025;
originally announced January 2025.
-
GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter
Authors:
Aniruddha Bala,
Rohan Jaiswal,
Loay Rashid,
Siddharth Roheda
Abstract:
Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, subopt…
▽ More
Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, suboptimal end-to-end models, data diversity constraints and high expenses. We propose an automated data generation pipeline aimed at alleviating such limitations, and introduce GalaxyEdit - a large-scale image editing dataset for add and remove operations. We fine-tune the SD v1.5 model on our dataset and find that our model can successfully handle a broader range of objects and complex editing instructions, outperforming state-of-the-art methods in FID scores by 11.2\% and 26.1\% for add and remove tasks respectively. Furthermore, in light of on-device usage scenarios, we expand our research to include task-specific lightweight adapters leveraging the ControlNet-xs architecture. While ControlNet-xs excels in canny and depth guided generation, we propose to improve the communication between the control network and U-Net for more intricate add and remove tasks. We achieve this by enhancing ControlNet-xs with non-linear interaction layers based on Volterra filters. Our approach outperforms ControlNet-xs in both add/remove and canny-guided image generation tasks, highlighting the effectiveness of the proposed enhancement.
△ Less
Submitted 20 November, 2024;
originally announced November 2024.
-
Scenic Routes over Points in 2D Space
Authors:
Loay Rashid,
Lini Thomas,
Kamalakar Karlapalem
Abstract:
Consider a 2D coordinate space with a set of red and a set of blue points. We define a scenic point as a point that is equidistant to a red point and a blue point. The set of contiguous scenic points form a scenic path. The perpendicular bisectors to the line joining a red point and a blue point forms a scenic path between the red point and the blue point. A scenic route is a traversal made from s…
▽ More
Consider a 2D coordinate space with a set of red and a set of blue points. We define a scenic point as a point that is equidistant to a red point and a blue point. The set of contiguous scenic points form a scenic path. The perpendicular bisectors to the line joining a red point and a blue point forms a scenic path between the red point and the blue point. A scenic route is a traversal made from scenic paths. In this paper, we address this novel problem by (i) designing algorithms for scenic route generation, (ii) studying the algorithms different properties and (iii) analyzing the routes generated by these algorithms. Scenic routes have applications in geo-spatial visualizations and visual analytics.
△ Less
Submitted 8 June, 2023;
originally announced June 2023.
