-
Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety
Authors:
Shashank Shriram,
Srinivasa Perisetla,
Aryan Keskar,
Harsha Krishnaswamy,
Tonko Emil Westerhof Bossen,
Andreas Møgelmose,
Ross Greer
Abstract:
Detecting anomalous hazards in visual data, particularly in video streams, is a critical challenge in autonomous driving. Existing models often struggle with unpredictable, out-of-label hazards due to their reliance on predefined object categories. In this paper, we propose a multimodal approach that integrates vision-language reasoning with zero-shot object detection to improve hazard identificat…
▽ More
Detecting anomalous hazards in visual data, particularly in video streams, is a critical challenge in autonomous driving. Existing models often struggle with unpredictable, out-of-label hazards due to their reliance on predefined object categories. In this paper, we propose a multimodal approach that integrates vision-language reasoning with zero-shot object detection to improve hazard identification and explanation. Our pipeline consists of a Vision-Language Model (VLM), a Large Language Model (LLM), in order to detect hazardous objects within a traffic scene. We refine object detection by incorporating OpenAI's CLIP model to match predicted hazards with bounding box annotations, improving localization accuracy. To assess model performance, we create a ground truth dataset by denoising and extending the foundational COOOL (Challenge-of-Out-of-Label) anomaly detection benchmark dataset with complete natural language descriptions for hazard annotations. We define a means of hazard detection and labeling evaluation on the extended dataset using cosine similarity. This evaluation considers the semantic similarity between the predicted hazard description and the annotated ground truth for each video. Additionally, we release a set of tools for structuring and managing large-scale hazard detection datasets. Our findings highlight the strengths and limitations of current vision-language-based approaches, offering insights into future improvements in autonomous hazard detection systems. Our models, scripts, and data can be found at https://github.com/mi3labucm/COOOLER.git
△ Less
Submitted 17 April, 2025;
originally announced April 2025.
-
doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigation
Authors:
Parthib Roy,
Srinivasa Perisetla,
Shashank Shriram,
Harsha Krishnaswamy,
Aryan Keskar,
Ross Greer
Abstract:
Human-interactive robotic systems, particularly autonomous vehicles (AVs), must effectively integrate human instructions into their motion planning. This paper introduces doScenes, a novel dataset designed to facilitate research on human-vehicle instruction interactions, focusing on short-term directives that directly influence vehicle motion. By annotating multimodal sensor data with natural lang…
▽ More
Human-interactive robotic systems, particularly autonomous vehicles (AVs), must effectively integrate human instructions into their motion planning. This paper introduces doScenes, a novel dataset designed to facilitate research on human-vehicle instruction interactions, focusing on short-term directives that directly influence vehicle motion. By annotating multimodal sensor data with natural language instructions and referentiality tags, doScenes bridges the gap between instruction and driving response, enabling context-aware and adaptive planning. Unlike existing datasets that focus on ranking or scene-level reasoning, doScenes emphasizes actionable directives tied to static and dynamic scene objects. This framework addresses limitations in prior research, such as reliance on simulated data or predefined action sets, by supporting nuanced and flexible responses in real-world scenarios. This work lays the foundation for developing learning strategies that seamlessly integrate human instructions into autonomous systems, advancing safe and effective human-vehicle collaboration for vision-language navigation. We make our data publicly available at https://www.github.com/rossgreer/doScenes
△ Less
Submitted 8 December, 2024;
originally announced December 2024.
-
Design and Testbed Deployment of Frequency-Domain Equalization Full Duplex Radios
Authors:
Manav Kohli,
Mahmood Baraani Dastjerdi,
Jin Zhou,
Ivan Seskar,
Harish Krishnaswamy,
Gil Zussman,
Tingjun Chen
Abstract:
Full-duplex (FD) wireless can significantly enhance spectrum efficiency but requires effective self-interference (SI) cancellers. RF SI cancellation (SIC) via frequency-domain equalization (FDE), where bandpass filters channelize the SI, is suited for integrated circuits (ICs). In this paper, we explore the limits and higher layer challenges associated with using such cancellers. We evaluate the p…
▽ More
Full-duplex (FD) wireless can significantly enhance spectrum efficiency but requires effective self-interference (SI) cancellers. RF SI cancellation (SIC) via frequency-domain equalization (FDE), where bandpass filters channelize the SI, is suited for integrated circuits (ICs). In this paper, we explore the limits and higher layer challenges associated with using such cancellers. We evaluate the performance of a custom FDE-based canceller using two testbeds; one with mobile FD radios and the other with upgraded, static FD radios in the PAWR COSMOS testbed. The latter is a lasting artifact for the research community, alongside a dataset containing baseband waveforms captured on the COSMOS FD radios, facilitating FD-related experimentation at the higher networking layers. We evaluate the performance of the FDE-based FD radios in both testbeds, with experiments showing 95 dB overall achieved SIC (52 dB from RF SIC) across 20 MHz bandwidth, and an average link-level FD rate gain of 1.87x. We also conduct experiments in (i) uplink-downlink networks with inter-user interference, and (ii) heterogeneous networks with half-duplex and FD users. The experimental FD gains in the two types of networks depend on the users' SNR values and the number of FD users, and are 1.14x-1.25x and 1.25x-1.73x, respectively, confirming previous analytical results.
△ Less
Submitted 31 January, 2024;
originally announced January 2024.
-
Wideband Full-Duplex Wireless via Frequency-Domain Equalization: Design and Experimentation
Authors:
Tingjun Chen,
Mahmood Baraani Dastjerdi,
Jin Zhou,
Harish Krishnaswamy,
Gil Zussman
Abstract:
Full-duplex (FD) wireless can significantly enhance spectrum efficiency but requires tremendous amount of self-interference (SI) cancellation. Recent advances in the RFIC community enabled wideband RF SI cancellation (SIC) in integrated circuits (ICs) via frequency-domain equalization (FDE), where RF filters channelize the SI signal path. Unlike other FD implementations, that mostly rely on delay…
▽ More
Full-duplex (FD) wireless can significantly enhance spectrum efficiency but requires tremendous amount of self-interference (SI) cancellation. Recent advances in the RFIC community enabled wideband RF SI cancellation (SIC) in integrated circuits (ICs) via frequency-domain equalization (FDE), where RF filters channelize the SI signal path. Unlike other FD implementations, that mostly rely on delay lines, FDE-based cancellers can be realized in small-form-factor devices. However, the fundamental limits and higher layer challenges associated with these cancellers were not explored yet. Therefore, and in order to support the integration with a software-defined radio (SDR) and to facilitate experimentation in a testbed with several nodes, we design and implement an FDE-based RF canceller on a printed circuit board (PCB). We derive and experimentally validate the PCB canceller model and present a canceller configuration scheme based on an optimization problem. We then extensively evaluate the performance of the FDE-based FD radio in the SDR testbed. Experiments show that it achieves 95dB overall SIC (52dB from RF SIC) across 20MHz bandwidth, and an average link-level FD gain of 1.87x. We also conduct experiments in: (i) uplink-downlink networks with inter-user interference, and (ii) heterogeneous networks with half-duplex and FD users. The experimental FD gains in the two types of networks confirm previous analytical results. They depend on the users' SNR values and the number of FD users, and are 1.14x-1.25x and 1.25x-1.73x, respectively. Finally, we numerically evaluate and compare the RFIC and PCB implementations and study various design tradeoffs.
△ Less
Submitted 3 December, 2018;
originally announced December 2018.
-
Open-Access Full-Duplex Wireless in the ORBIT Testbed
Authors:
Tingjun Chen,
Mahmood Baraani Dastjerdi,
Guy Farkash,
Jin Zhou,
Harish Krishnaswamy,
Gil Zussman
Abstract:
In order to support experimentation with full-duplex (FD) wireless, we recently integrated an open-access FD transceiver in the ORBIT testbed. In this report, we present the design and implementation of the FD transceiver and interfaces, and provide examples and guidelines for experimentation. In particular, an ORBIT node with a National Instruments (NI)/Ettus Research Universal Software Radio Per…
▽ More
In order to support experimentation with full-duplex (FD) wireless, we recently integrated an open-access FD transceiver in the ORBIT testbed. In this report, we present the design and implementation of the FD transceiver and interfaces, and provide examples and guidelines for experimentation. In particular, an ORBIT node with a National Instruments (NI)/Ettus Research Universal Software Radio Peripheral (USRP) N210 software-defined radio (SDR) was equipped with the Columbia FlexICoN Gen-1 customized RF self-interference (SI) canceller box. The RF canceller box includes an RF SI canceller that is implemented using discrete components on a printed circuit board (PCB) and achieves 40dB RF SI cancellation across 5MHz bandwidth. We provide an FD transceiver baseline program and present two example FD experiments where 90dB and 85dB overall SI cancellation is achieved for a simple waveform and PSK modulated signals across both the RF and digital domains. We also discuss potential FD wireless experiments that can be conducted based on the implemented open-access FD transceiver and baseline program.
△ Less
Submitted 29 May, 2018; v1 submitted 9 January, 2018;
originally announced January 2018.
-
Resource Allocation and Rate Gains in Practical Full-Duplex Systems
Authors:
Jelena Marašević,
Jin Zhou,
Harish Krishnaswamy,
Yuan Zhong,
Gil Zussman
Abstract:
Full-duplex communication has the potential to substantially increase the throughput in wireless networks. However, the benefits of full-duplex are still not well understood. In this paper, we characterize the full-duplex rate gains in both single-channel and multi-channel use cases. For the single-channel case, we quantify the rate gain as a function of the remaining self-interference and SNR val…
▽ More
Full-duplex communication has the potential to substantially increase the throughput in wireless networks. However, the benefits of full-duplex are still not well understood. In this paper, we characterize the full-duplex rate gains in both single-channel and multi-channel use cases. For the single-channel case, we quantify the rate gain as a function of the remaining self-interference and SNR values. We also provide a sufficient condition under which the sum of uplink and downlink rates on a full-duplex channel is concave in the transmission power levels. Building on these results, we consider the multi-channel case. For that case, we introduce a new realistic model of a small form-factor (e.g., smartphone) full-duplex receiver and demonstrate its accuracy via measurements. We study the problem of jointly allocating power levels to different channels and selecting the frequency of maximum self-interference suppression, where the objective is maximizing the sum of the rates over uplink and downlink OFDM channels. We develop a polynomial time algorithm which is nearly optimal in practice under very mild restrictions. To reduce the running time, we develop an efficient nearly-optimal algorithm under the high SINR approximation. Finally, we demonstrate via numerical evaluations the capacity gains in the different use cases and obtain insights into the impact of the remaining self-interference and wireless channel states on the performance.
△ Less
Submitted 8 June, 2016; v1 submitted 27 March, 2015;
originally announced March 2015.
