Showing 1–6 of 6 results for author: Jawaid, M

Event-RGB Fusion for Spacecraft Pose Estimation Under Harsh Lighting

Authors: Mohsi Jawaid, Marcus Märtens, Tat-Jun Chin

Abstract: Spacecraft pose estimation is crucial for autonomous in-space operations, such as rendezvous, docking and on-orbit servicing. Vision-based pose estimation methods, which typically employ RGB imaging sensors, is a compelling solution for spacecraft pose estimation, but are challenged by harsh lighting conditions, which produce imaging artifacts such as glare, over-exposure, blooming and lens flare.… ▽ More Spacecraft pose estimation is crucial for autonomous in-space operations, such as rendezvous, docking and on-orbit servicing. Vision-based pose estimation methods, which typically employ RGB imaging sensors, is a compelling solution for spacecraft pose estimation, but are challenged by harsh lighting conditions, which produce imaging artifacts such as glare, over-exposure, blooming and lens flare. Due to their much higher dynamic range, neuromorphic or event sensors are more resilient to extreme lighting conditions. However, event sensors generally have lower spatial resolution and suffer from reduced signal-to-noise ratio during periods of low relative motion. This work addresses these individual sensor limitations by introducing a sensor fusion approach combining RGB and event sensors. A beam-splitter prism was employed to achieve precise optical and temporal alignment. Then, a RANSAC-based technique was developed to fuse the information from the RGB and event channels to achieve pose estimation that leveraged the strengths of the two modalities. The pipeline was complemented by dropout uncertainty estimation to detect extreme conditions that affect either channel. To benchmark the performance of the proposed event-RGB fusion method, we collected a comprehensive real dataset of RGB and event data for satellite pose estimation in a laboratory setting under a variety of challenging illumination conditions. Encouraging results on the dataset demonstrate the efficacy of our event-RGB fusion approach and further supports the usage of event sensors for spacecraft pose estimation. To support community research on this topic, our dataset will be released publicly. △ Less

Submitted 8 July, 2025; originally announced July 2025.

Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimation

Authors: Mohsi Jawaid, Rajat Talak, Yasir Latif, Luca Carlone, Tat-Jun Chin

Abstract: Deep learning plays a critical role in vision-based satellite pose estimation. However, the scarcity of real data from the space environment means that deep models need to be trained using synthetic data, which raises the Sim2Real domain gap problem. A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. Event sensors have been shown to provide some robustnes… ▽ More Deep learning plays a critical role in vision-based satellite pose estimation. However, the scarcity of real data from the space environment means that deep models need to be trained using synthetic data, which raises the Sim2Real domain gap problem. A major cause of the Sim2Real gap are novel lighting conditions encountered during test time. Event sensors have been shown to provide some robustness against lighting variations in vision-based pose estimation. However, challenging lighting conditions due to strong directional light can still cause undesirable effects in the output of commercial off-the-shelf event sensors, such as noisy/spurious events and inhomogeneous event densities on the object. Such effects are non-trivial to simulate in software, thus leading to Sim2Real gap in the event domain. To close the Sim2Real gap in event-based satellite pose estimation, the paper proposes a test-time self-supervision scheme with a certifier module. Self-supervision is enabled by an optimisation routine that aligns a dense point cloud of the predicted satellite pose with the event data to attempt to rectify the inaccurately estimated pose. The certifier attempts to verify the corrected pose, and only certified test-time inputs are backpropagated via implicit differentiation to refine the predicted landmarks, thus improving the pose estimates and closing the Sim2Real gap. Results show that the our method outperforms established test-time adaptation schemes. △ Less

Submitted 10 September, 2024; originally announced September 2024.

Comments: This work has been accepted for publication at IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)

Improving few-shot learning-based protein engineering with evolutionary sampling

Authors: M. Zaki Jawaid, Robin W. Yeo, Aayushma Gautam, T. Blair Gainous, Daniel O. Hart, Timothy P. Daley

Abstract: Designing novel functional proteins remains a slow and expensive process due to a variety of protein engineering challenges; in particular, the number of protein variants that can be experimentally tested in a given assay pales in comparison to the vastness of the overall sequence space, resulting in low hit rates and expensive wet lab testing cycles. In this paper, we propose a few-shot learning… ▽ More Designing novel functional proteins remains a slow and expensive process due to a variety of protein engineering challenges; in particular, the number of protein variants that can be experimentally tested in a given assay pales in comparison to the vastness of the overall sequence space, resulting in low hit rates and expensive wet lab testing cycles. In this paper, we propose a few-shot learning approach to novel protein design that aims to accelerate the expensive wet lab testing cycle and is capable of leveraging a training dataset that is both small and skewed ($\approx 10^5$ datapoints, $< 1\%$ positive hits). Our approach is composed of two parts: a semi-supervised transfer learning approach to generate a discrete fitness landscape for a desired protein function and a novel evolutionary Monte Carlo Markov Chain sampling algorithm to more efficiently explore the fitness landscape. We demonstrate the performance of our approach by experimentally screening predicted high fitness gene activators, resulting in a dramatically improved hit rate compared to existing methods. Our method can be easily adapted to other protein engineering and design problems, particularly where the cost associated with obtaining labeled data is significantly high. We have provided open source code for our method at https:// github.com/SuperSecretBioTech/evolutionary_monte_carlo_search. △ Less

Submitted 25 May, 2023; v1 submitted 23 May, 2023; originally announced May 2023.

Towards Bridging the Space Domain Gap for Satellite Pose Estimation using Event Sensing

Authors: Mohsi Jawaid, Ethan Elms, Yasir Latif, Tat-Jun Chin

Abstract: Deep models trained using synthetic data require domain adaptation to bridge the gap between the simulation and target environments. State-of-the-art domain adaptation methods often demand sufficient amounts of (unlabelled) data from the target domain. However, this need is difficult to fulfil when the target domain is an extreme environment, such as space. In this paper, our target problem is clo… ▽ More Deep models trained using synthetic data require domain adaptation to bridge the gap between the simulation and target environments. State-of-the-art domain adaptation methods often demand sufficient amounts of (unlabelled) data from the target domain. However, this need is difficult to fulfil when the target domain is an extreme environment, such as space. In this paper, our target problem is close proximity satellite pose estimation, where it is costly to obtain images of satellites from actual rendezvous missions. We demonstrate that event sensing offers a promising solution to generalise from the simulation to the target domain under stark illumination differences. Our main contribution is an event-based satellite pose estimation technique, trained purely on synthetic event data with basic data augmentation to improve robustness against practical (noisy) event sensors. Underpinning our method is a novel dataset with carefully calibrated ground truth, comprising of real event data obtained by emulating satellite rendezvous scenarios in the lab under drastic lighting conditions. Results on the dataset showed that our event-based satellite pose estimation method, trained only on synthetic data without adaptation, could generalise to the target domain effectively. △ Less

Submitted 24 September, 2022; originally announced September 2022.

Comments: 8 pages. This work has been submitted to the IEEE (ICRA 2023) for possible publication

G2D: from GTA to Data

Authors: Anh-Dzung Doan, Abdul Mohsi Jawaid, Thanh-Toan Do, Tat-Jun Chin

Abstract: This document describes G2D, a software that enables capturing videos from Grand Theft Auto V (GTA V), a popular role playing game set in an expansive virtual city. The target users of our software are computer vision researchers who wish to collect hyper-realistic computer-generated imagery of a city from the street level, under controlled 6DOF camera poses and varying environmental conditions (w… ▽ More This document describes G2D, a software that enables capturing videos from Grand Theft Auto V (GTA V), a popular role playing game set in an expansive virtual city. The target users of our software are computer vision researchers who wish to collect hyper-realistic computer-generated imagery of a city from the street level, under controlled 6DOF camera poses and varying environmental conditions (weather, season, time of day, traffic density, etc.). G2D accesses/calls the native functions of the game; hence users can directly interact with G2D while playing the game. Specifically, G2D enables users to manipulate conditions of the virtual environment on the fly, while the gameplay camera is set to automatically retrace a predetermined 6DOF camera pose trajectory within the game coordinate system. Concurrently, automatic screen capture is executed while the virtual environment is being explored. G2D and its source code are publicly available at https://goo.gl/SS7fS6 In addition, we demonstrate an application of G2D to generate a large-scale dataset with groundtruth camera poses for testing structure-from-motion (SfM) algorithms. The dataset and generated 3D point clouds are also made available at https://goo.gl/DNzxHx △ Less

Submitted 16 June, 2018; originally announced June 2018.

Comments: 9 pages, 6 figures, 2 tables

Segmentation of Soft atherosclerotic plaques using active contour models

Authors: Muhammad Moazzam Jawaid

Abstract: Detection of non-calcified plaques in the coronary tree is a challenging problem due to the nature of comprising substances. Hard plaques are easily discernible in CTA data cloud due to apparent bright behaviour, therefore many approaches have been proposed for automatic segmentation of calcified plaques. In contrast soft plaques show very small difference in intensity with respect to surrounding… ▽ More Detection of non-calcified plaques in the coronary tree is a challenging problem due to the nature of comprising substances. Hard plaques are easily discernible in CTA data cloud due to apparent bright behaviour, therefore many approaches have been proposed for automatic segmentation of calcified plaques. In contrast soft plaques show very small difference in intensity with respect to surrounding heart tissues & blood voxels. This similarity in intensity makes the isolation and detection of soft plaques very difficult. This work aims to develop framework for segmentation of vulnerable plaques with minimal user dependency. In first step automatic seed point has been established based on the fact that coronary artery behaves as tubular structure through axial slices. In the following step the behaviour of contrast agent has been modelled mathematically to reflect the dye diffusion in respective CTA volume. Consequently based on detected seed point & intensity behaviour, localized active contour segmentation has been applied to extract complete coronary tree. Bidirectional segmentation has been applied to avoid loss of coronary information due to the seed point location whereas auto adjustment feature of contour grabs new emerging branches. Medial axis for extracted coronary tree is generated using fast marching method for obtaining curve planar reformation for validation of contrast agent behaviour. Obtained coronary tree is to be evaluated for soft plaques in second phase of this research. △ Less

Submitted 30 July, 2016; originally announced August 2016.