-
From Words to Poses: Enhancing Novel Object Pose Estimation with Vision Language Models
Authors:
Tessa Pulli,
Stefan Thalhammer,
Simon Schwaiger,
Markus Vincze
Abstract:
Robots are increasingly envisioned to interact in real-world scenarios, where they must continuously adapt to new situations. To detect and grasp novel objects, zero-shot pose estimators determine poses without prior knowledge. Recently, vision language models (VLMs) have shown considerable advances in robotics applications by establishing an understanding between language input and image input. I…
▽ More
Robots are increasingly envisioned to interact in real-world scenarios, where they must continuously adapt to new situations. To detect and grasp novel objects, zero-shot pose estimators determine poses without prior knowledge. Recently, vision language models (VLMs) have shown considerable advances in robotics applications by establishing an understanding between language input and image input. In our work, we take advantage of VLMs zero-shot capabilities and translate this ability to 6D object pose estimation. We propose a novel framework for promptable zero-shot 6D object pose estimation using language embeddings. The idea is to derive a coarse location of an object based on the relevancy map of a language-embedded NeRF reconstruction and to compute the pose estimate with a point cloud registration method. Additionally, we provide an analysis of LERF's suitability for open-set object pose estimation. We examine hyperparameters, such as activation thresholds for relevancy maps and investigate the zero-shot capabilities on an instance- and category-level. Furthermore, we plan to conduct robotic grasping experiments in a real-world setting.
△ Less
Submitted 9 September, 2024;
originally announced September 2024.
-
UGV-CBRN: An Unmanned Ground Vehicle for Chemical, Biological, Radiological, and Nuclear Disaster Response
Authors:
Simon Schwaiger,
Lucas Muster,
Georg Novotny,
Michael Schebek,
Wilfried Wöber,
Stefan Thalhammer,
Christoph Böhm
Abstract:
Robotic search and rescue (SAR) supports response teams by accelerating disaster assessment and by keeping operators away from hazardous environments. In the event of a chemical, biological, radiological, and nuclear (CBRN) disaster, robots are deployed to identify and locate radiation sources. Human responders then assess the situation and neutralize the danger. The presented system takes a step…
▽ More
Robotic search and rescue (SAR) supports response teams by accelerating disaster assessment and by keeping operators away from hazardous environments. In the event of a chemical, biological, radiological, and nuclear (CBRN) disaster, robots are deployed to identify and locate radiation sources. Human responders then assess the situation and neutralize the danger. The presented system takes a step toward enhanced integration of robots into SAR teams. Integrating autonomous radiation mapping with semi-autonomous substance sampling and online analysis of the CBRN threat lets the human operator localize and assess the threat from a safe distance. Two LiDARs, an IMU, and a Geiger counter are used for mapping the surrounding area and localizing potential radiation sources. A mobile manipulator with six Degrees of Freedom manipulates valves and samples substances that are analyzed by an onboard Raman spectrometer. The human operator monitors the mission's progression from a remote location defining target locations and directing the semi-autonomous manipulation processes. Diverse recovery behaviours aid robot deployment, system state monitoring, as well as recovery of hard- and software. Field tests showcase the capabilities of the presented system during trials at the CBRN disaster response challenge European Robotics Hackathon (EnRicH). We provide recorded sensor data and implemented software through a GitHub repository: https://github.com/TW-Robotics/search-and-rescue-robot-2024.
△ Less
Submitted 20 September, 2024; v1 submitted 20 June, 2024;
originally announced June 2024.
-
Fully Automatic Page Turning on Real Scores
Authors:
Florian Henkel,
Stephanie Schwaiger,
Gerhard Widmer
Abstract:
We present a prototype of an automatic page turning system that works directly on real scores, i.e., sheet images, without any symbolic representation. Our system is based on a multi-modal neural network architecture that observes a complete sheet image page as input, listens to an incoming musical performance, and predicts the corresponding position in the image. Using the position estimation of…
▽ More
We present a prototype of an automatic page turning system that works directly on real scores, i.e., sheet images, without any symbolic representation. Our system is based on a multi-modal neural network architecture that observes a complete sheet image page as input, listens to an incoming musical performance, and predicts the corresponding position in the image. Using the position estimation of our system, we use a simple heuristic to trigger a page turning event once a certain location within the sheet image is reached. As a proof of concept we further combine our system with an actual machine that will physically turn the page on command.
△ Less
Submitted 12 November, 2021;
originally announced November 2021.
-
Enhanced Transmission in Rolled-up Hyperlenses utilizing Fabry-Peŕot Resonances
Authors:
Jochen Kerbst,
Stephan Schwaiger,
Andreas Rottler,
Aune Koitmäe,
Markus Bröll,
Andrea Stemmann,
Christian Heyn,
Detlef Heitmann,
Stefan Mendach
Abstract:
We experimentally demonstrate that the transmission though rolled-up metal/semiconductor hyperlenses can be enhanced at desired frequencies utilizing Fabry-Pérot resonances. By means of finite difference time domain simulations we prove that hyperlensing occurs at frequencies of high transmission.
We experimentally demonstrate that the transmission though rolled-up metal/semiconductor hyperlenses can be enhanced at desired frequencies utilizing Fabry-Pérot resonances. By means of finite difference time domain simulations we prove that hyperlensing occurs at frequencies of high transmission.
△ Less
Submitted 8 July, 2011;
originally announced July 2011.
-
Transmission enhancement in three-dimensional rolled-up plasmonic metamaterials containing optically active quantum wells
Authors:
Andreas Rottler,
Stephan Schwaiger,
Anuen Koitmäe,
Detlef Heitmann,
Stefan Mendach
Abstract:
We investigate three-dimensional rolled-up metamaterials containing optically active quantum wells and metal gratings supporting surface plasmon polarition resonances. Finite-difference time-domain simulations show that by matching the surface plasmon polarition resonance with the active wavelength regime of the quantum well a strong transmission enhancement is observed when illuminating the sampl…
▽ More
We investigate three-dimensional rolled-up metamaterials containing optically active quantum wells and metal gratings supporting surface plasmon polarition resonances. Finite-difference time-domain simulations show that by matching the surface plasmon polarition resonance with the active wavelength regime of the quantum well a strong transmission enhancement is observed when illuminating the sample with p-polarized radiation. This transmission enhancement is further increased by taking advantage of the Fabry-Perot resonances of the structure.
△ Less
Submitted 23 May, 2011;
originally announced May 2011.
-
Gain in Three-Dimensional Metamaterials utilizing Semiconductor Quantum Structures
Authors:
Stephan Schwaiger,
Matthias Klingbeil,
Jochen Kerbst,
Andreas Rottler,
Ricardo Costa,
Aune Koitmäe,
Markus Bröll,
Christian Heyn,
Yuliya Stark,
Detlef Heitmann,
Stefan Mendach
Abstract:
We demonstrate gain in a three-dimensional metal/semiconductor metamaterial by the integration of optically active semiconductor quantum structures. The rolling-up of a metallic structure on top of strained semiconductor layers containing a quantum well allows us to achieve a three-dimensional superlattice consisting of alternating layers of lossy metallic and amplifying gain material. We show tha…
▽ More
We demonstrate gain in a three-dimensional metal/semiconductor metamaterial by the integration of optically active semiconductor quantum structures. The rolling-up of a metallic structure on top of strained semiconductor layers containing a quantum well allows us to achieve a three-dimensional superlattice consisting of alternating layers of lossy metallic and amplifying gain material. We show that the transmission through the superlattice can be enhanced by exciting the quantum well optically under both pulsed or continuous wave excitation. This points out that our structures can be used as a starting point for arbitrary three-dimensional metamaterials including gain.
△ Less
Submitted 12 April, 2011;
originally announced April 2011.
-
Magnetic anisotropy in (Ga,Mn)As: Influence of epitaxial strain and hole concentration
Authors:
M. Glunk,
J. Daeubler,
L. Dreher,
S. Schwaiger,
W. Schoch,
R. Sauer,
W. Limmer,
A. Brandlmaier,
S. T. B. Goennenwein,
C. Bihler,
M. S. Brandt
Abstract:
We present a systematic study on the influence of epitaxial strain and hole concentration on the magnetic anisotropy in (Ga,Mn)As at 4.2 K. The strain was gradually varied over a wide range from tensile to compressive by growing a series of (Ga,Mn)As layers with 5% Mn on relaxed graded (In,Ga)As/GaAs templates with different In concentration. The hole density, the Curie temperature, and the rela…
▽ More
We present a systematic study on the influence of epitaxial strain and hole concentration on the magnetic anisotropy in (Ga,Mn)As at 4.2 K. The strain was gradually varied over a wide range from tensile to compressive by growing a series of (Ga,Mn)As layers with 5% Mn on relaxed graded (In,Ga)As/GaAs templates with different In concentration. The hole density, the Curie temperature, and the relaxed lattice constant of the as-grown and annealed (Ga,Mn)As layers turned out to be essentially unaffected by the strain. Angle-dependent magnetotransport measurements performed at different magnetic field strengths were used to probe the magnetic anisotropy. The measurements reveal a pronounced linear dependence of the uniaxial out-of-plane anisotropy on both strain and hole density. Whereas the uniaxial and cubic in-plane anisotropies are nearly constant, the cubic out-of-plane anisotropy changes sign when the magnetic easy axis flips from in-plane to out-of-plane. The experimental results for the magnetic anisotropy are quantitatively compared with calculations of the free energy based on a mean-field Zener model. An almost perfect agreement between experiment and theory is found for the uniaxial out-of-plane and cubic in-plane anisotropy parameters of the as-grown samples. In addition, magnetostriction constants are derived from the anisotropy data.
△ Less
Submitted 9 April, 2009;
originally announced April 2009.
-
Advanced resistivity model for arbitrary magnetization orientation applied to a series of compressive- to tensile-strained (Ga,Mn)As layers
Authors:
W. Limmer,
J. Daeubler,
L. Dreher,
M. Glunk,
W. Schoch,
S. Schwaiger,
R. Sauer
Abstract:
The longitudinal and transverse resistivities of differently strained (Ga,Mn)As layers are theoretically and experimentally studied as a function of the magnetization orientation. The strain in the series of (Ga,Mn)As layers is gradually varied from compressive to tensile using (In,Ga)As templates with different In concentrations. Analytical expressions for the resistivities are derived from a s…
▽ More
The longitudinal and transverse resistivities of differently strained (Ga,Mn)As layers are theoretically and experimentally studied as a function of the magnetization orientation. The strain in the series of (Ga,Mn)As layers is gradually varied from compressive to tensile using (In,Ga)As templates with different In concentrations. Analytical expressions for the resistivities are derived from a series expansion of the resistivity tensor with respect to the direction cosines of the magnetization. In order to quantitatively model the experimental data, terms up to the fourth order have to be included. The expressions derived are generally valid for any single-crystalline cubic and tetragonal ferromagnet and apply to arbitrary surface orientations and current directions. The model phenomenologically incorporates the longitudinal and transverse anisotropic magnetoresistance as well as the anomalous Hall effect. The resistivity parameters obtained from a comparison between experiment and theory are found to systematically vary with the strain in the layer.
△ Less
Submitted 19 February, 2008;
originally announced February 2008.
