Showing 1–17 of 17 results for author: Pfrommer, J

6D Pose Estimation on Point Cloud Data through Prior Knowledge Integration: A Case Study in Autonomous Disassembly

Authors: Chengzhi Wu, Hao Fu, Jan-Philipp Kaiser, Erik Tabuchi Barczak, Julius Pfrommer, Gisela Lanza, Michael Heizmann, Jürgen Beyerer

Abstract: The accurate estimation of 6D pose remains a challenging task within the computer vision domain, even when utilizing 3D point cloud data. Conversely, in the manufacturing domain, instances arise where leveraging prior knowledge can yield advancements in this endeavor. This study focuses on the disassembly of starter motors to augment the engineering of product life cycles. A pivotal objective in t… ▽ More The accurate estimation of 6D pose remains a challenging task within the computer vision domain, even when utilizing 3D point cloud data. Conversely, in the manufacturing domain, instances arise where leveraging prior knowledge can yield advancements in this endeavor. This study focuses on the disassembly of starter motors to augment the engineering of product life cycles. A pivotal objective in this context involves the identification and 6D pose estimation of bolts affixed to the motors, facilitating automated disassembly within the manufacturing workflow. Complicating matters, the presence of occlusions and the limitations of single-view data acquisition, notably when motors are placed in a clamping system, obscure certain portions and render some bolts imperceptible. Consequently, the development of a comprehensive pipeline capable of acquiring complete bolt information is imperative to avoid oversight in bolt detection. In this paper, employing the task of bolt detection within the scope of our project as a pertinent use case, we introduce a meticulously devised pipeline. This multi-stage pipeline effectively captures the 6D information with regard to all bolts on the motor, thereby showcasing the effective utilization of prior knowledge in handling this challenging task. The proposed methodology not only contributes to the field of 6D pose estimation but also underscores the viability of integrating domain-specific insights to tackle complex problems in manufacturing and automation. △ Less

Submitted 30 May, 2025; originally announced May 2025.

A Cross Branch Fusion-Based Contrastive Learning Framework for Point Cloud Self-supervised Learning

Authors: Chengzhi Wu, Qianliang Huang, Kun Jin, Julius Pfrommer, Jürgen Beyerer

Abstract: Contrastive learning is an essential method in self-supervised learning. It primarily employs a multi-branch strategy to compare latent representations obtained from different branches and train the encoder. In the case of multi-modal input, diverse modalities of the same object are fed into distinct branches. When using single-modal data, the same input undergoes various augmentations before bein… ▽ More Contrastive learning is an essential method in self-supervised learning. It primarily employs a multi-branch strategy to compare latent representations obtained from different branches and train the encoder. In the case of multi-modal input, diverse modalities of the same object are fed into distinct branches. When using single-modal data, the same input undergoes various augmentations before being fed into different branches. However, all existing contrastive learning frameworks have so far only performed contrastive operations on the learned features at the final loss end, with no information exchange between different branches prior to this stage. In this paper, for point cloud unsupervised learning without the use of extra training data, we propose a Contrastive Cross-branch Attention-based framework for Point cloud data (termed PoCCA), to learn rich 3D point cloud representations. By introducing sub-branches, PoCCA allows information exchange between different branches before the loss end. Experimental results demonstrate that in the case of using no extra training data, the representations learned with our self-supervised model achieve state-of-the-art performances when used for downstream tasks on point clouds. △ Less

Submitted 30 May, 2025; originally announced May 2025.

SAMBLE: Shape-Specific Point Cloud Sampling for an Optimal Trade-Off Between Local Detail and Global Uniformity

Authors: Chengzhi Wu, Yuxin Wan, Hao Fu, Julius Pfrommer, Zeyun Zhong, Junwei Zheng, Jiaming Zhang, Jürgen Beyerer

Abstract: Driven by the increasing demand for accurate and efficient representation of 3D data in various domains, point cloud sampling has emerged as a pivotal research topic in 3D computer vision. Recently, learning-to-sample methods have garnered growing interest from the community, particularly for their ability to be jointly trained with downstream tasks. However, previous learning-based sampling metho… ▽ More Driven by the increasing demand for accurate and efficient representation of 3D data in various domains, point cloud sampling has emerged as a pivotal research topic in 3D computer vision. Recently, learning-to-sample methods have garnered growing interest from the community, particularly for their ability to be jointly trained with downstream tasks. However, previous learning-based sampling methods either lead to unrecognizable sampling patterns by generating a new point cloud or biased sampled results by focusing excessively on sharp edge details. Moreover, they all overlook the natural variations in point distribution across different shapes, applying a similar sampling strategy to all point clouds. In this paper, we propose a Sparse Attention Map and Bin-based Learning method (termed SAMBLE) to learn shape-specific sampling strategies for point cloud shapes. SAMBLE effectively achieves an improved balance between sampling edge points for local details and preserving uniformity in the global shape, resulting in superior performance across multiple common point cloud downstream tasks, even in scenarios with few-point sampling. △ Less

Submitted 30 May, 2025; v1 submitted 28 April, 2025; originally announced April 2025.

Partially Observable Gaussian Process Network and Doubly Stochastic Variational Inference

Authors: Saksham Kiroriwal, Julius Pfrommer, Jürgen Beyerer

Abstract: To reduce the curse of dimensionality for Gaussian processes (GP), they can be decomposed into a Gaussian Process Network (GPN) of coupled subprocesses with lower dimensionality. In some cases, intermediate observations are available within the GPN. However, intermediate observations are often indirect, noisy, and incomplete in most real-world systems. This work introduces the Partially Observable… ▽ More To reduce the curse of dimensionality for Gaussian processes (GP), they can be decomposed into a Gaussian Process Network (GPN) of coupled subprocesses with lower dimensionality. In some cases, intermediate observations are available within the GPN. However, intermediate observations are often indirect, noisy, and incomplete in most real-world systems. This work introduces the Partially Observable Gaussian Process Network (POGPN) to model real-world process networks. We model a joint distribution of latent functions of subprocesses and make inferences using observations from all subprocesses. POGPN incorporates observation lenses (observation likelihoods) into the well-established inference method of deep Gaussian processes. We also introduce two training methods for POPGN to make inferences on the whole network using node observations. The application to benchmark problems demonstrates how incorporating partial observations during training and inference can improve the predictive performance of the overall network, offering a promising outlook for its practical application. △ Less

Submitted 19 February, 2025; originally announced February 2025.

Comments: 8 pages, 6 figures

Rethinking Attention Module Design for Point Cloud Analysis

Authors: Chengzhi Wu, Kaige Wang, Zeyun Zhong, Hao Fu, Junwei Zheng, Jiaming Zhang, Julius Pfrommer, Jürgen Beyerer

Abstract: In recent years, there have been significant advancements in applying attention mechanisms to point cloud analysis. However, attention module variants featured in various research papers often operate under diverse settings and tasks, incorporating potential training strategies. This heterogeneity poses challenges in establishing a fair comparison among these attention module variants. In this pap… ▽ More In recent years, there have been significant advancements in applying attention mechanisms to point cloud analysis. However, attention module variants featured in various research papers often operate under diverse settings and tasks, incorporating potential training strategies. This heterogeneity poses challenges in establishing a fair comparison among these attention module variants. In this paper, we address this issue by rethinking and exploring attention module design within a consistent base framework and settings. Both global-based and local-based attention methods are studied, with a focus on the selection basis and scales of neighbors for local-based attention. Different combinations of aggregated local features and computation methods for attention scores are evaluated, ranging from the initial addition/concatenation-based approach to the widely adopted dot product-based method and the recently proposed vector attention technique. Various position encoding methods are also investigated. Our extensive experimental analysis reveals that there is no universally optimal design across diverse point cloud tasks. Instead, drawing from best practices, we propose tailored attention modules for specific tasks, leading to superior performance on point cloud classification and segmentation benchmarks. △ Less

Submitted 27 July, 2024; originally announced July 2024.

Sorting multibay block stacking storage systems

Authors: Jakob Pfrommer, Thomas Bömer, Daniyar Akizhanov, Anne Meyer

Abstract: Autonomous mobile robots (AMRs) are increasingly used to automate operations in intralogistics. One crucial feature of AMRs is their availability, allowing them to operate 24/7. This work addresses the multibay unit load pre-marshalling problem, which extends pre-marshalling from a single bay to larger warehouse configurations with multiple bays. Pre-marshalling leverages off-peak time intervals t… ▽ More Autonomous mobile robots (AMRs) are increasingly used to automate operations in intralogistics. One crucial feature of AMRs is their availability, allowing them to operate 24/7. This work addresses the multibay unit load pre-marshalling problem, which extends pre-marshalling from a single bay to larger warehouse configurations with multiple bays. Pre-marshalling leverages off-peak time intervals to sort a block stacking warehouse in anticipation of future orders. These larger warehouse configurations require not only the minimization of the number of moves but also the consideration of distance or time when making sorting decisions. Our proposed solution for the multibay unit load pre-marshalling problem is based on our two-step approach that first determines the access direction for each stack and then finds a sequence of moves to sort the warehouse. In addition to adapting the existing approach that integrates a network flow model and an extended A* algorithm, we additionally present an exact constraint programming approach for the second stage of the problem-solving process. The results demonstrate that the presented solution approach effectively enhances the access time of unit loads and reduces the sorting effort for block stacking warehouses with multiple bays. △ Less

Submitted 8 May, 2024; originally announced May 2024.

Attention-based Part Assembly for 3D Volumetric Shape Modeling

Authors: Chengzhi Wu, Junwei Zheng, Julius Pfrommer, Jürgen Beyerer

Abstract: Modeling a 3D volumetric shape as an assembly of decomposed shape parts is much more challenging, but semantically more valuable than direct reconstruction from a full shape representation. The neural network needs to implicitly learn part relations coherently, which is typically performed by dedicated network layers that can generate transformation matrices for each part. In this paper, we propos… ▽ More Modeling a 3D volumetric shape as an assembly of decomposed shape parts is much more challenging, but semantically more valuable than direct reconstruction from a full shape representation. The neural network needs to implicitly learn part relations coherently, which is typically performed by dedicated network layers that can generate transformation matrices for each part. In this paper, we propose a VoxAttention network architecture for attention-based part assembly. We further propose a variant of using channel-wise part attention and show the advantages of this approach. Experimental results show that our method outperforms most state-of-the-art methods for the part relation-aware 3D shape modeling task. △ Less

Submitted 17 April, 2023; originally announced April 2023.

Attention-based Point Cloud Edge Sampling

Authors: Chengzhi Wu, Junwei Zheng, Julius Pfrommer, Jürgen Beyerer

Abstract: Point cloud sampling is a less explored research topic for this data representation. The most commonly used sampling methods are still classical random sampling and farthest point sampling. With the development of neural networks, various methods have been proposed to sample point clouds in a task-based learning manner. However, these methods are mostly generative-based, rather than selecting poin… ▽ More Point cloud sampling is a less explored research topic for this data representation. The most commonly used sampling methods are still classical random sampling and farthest point sampling. With the development of neural networks, various methods have been proposed to sample point clouds in a task-based learning manner. However, these methods are mostly generative-based, rather than selecting points directly using mathematical statistics. Inspired by the Canny edge detection algorithm for images and with the help of the attention mechanism, this paper proposes a non-generative Attention-based Point cloud Edge Sampling method (APES), which captures salient points in the point cloud outline. Both qualitative and quantitative experimental results show the superior performance of our sampling method on common benchmark tasks. △ Less

Submitted 26 March, 2023; v1 submitted 28 February, 2023; originally announced February 2023.

Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly

Authors: Chengzhi Wu, Xuelei Bi, Julius Pfrommer, Alexander Cebulla, Simon Mangold, Jürgen Beyerer

Abstract: On robotics computer vision tasks, generating and annotating large amounts of data from real-world for the use of deep learning-based approaches is often difficult or even impossible. A common strategy for solving this problem is to apply simulation-to-reality (sim2real) approaches with the help of simulated scenes. While the majority of current robotics vision sim2real work focuses on image data,… ▽ More On robotics computer vision tasks, generating and annotating large amounts of data from real-world for the use of deep learning-based approaches is often difficult or even impossible. A common strategy for solving this problem is to apply simulation-to-reality (sim2real) approaches with the help of simulated scenes. While the majority of current robotics vision sim2real work focuses on image data, we present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data in order to achieve better performance when the learned model is transferred to real-world data. The issue of imbalanced learning is investigated using multiple strategies. A novel patch-based attention network is proposed additionally to tackle this problem. △ Less

Submitted 12 January, 2023; originally announced January 2023.

MotorFactory: A Blender Add-on for Large Dataset Generation of Small Electric Motors

Authors: Chengzhi Wu, Kanran Zhou, Jan-Philipp Kaiser, Norbert Mitschke, Jan-Felix Klein, Julius Pfrommer, Jürgen Beyerer, Gisela Lanza, Michael Heizmann, Kai Furmans

Abstract: To enable automatic disassembly of different product types with uncertain conditions and degrees of wear in remanufacturing, agile production systems that can adapt dynamically to changing requirements are needed. Machine learning algorithms can be employed due to their generalization capabilities of learning from various types and variants of products. However, in reality, datasets with a diversi… ▽ More To enable automatic disassembly of different product types with uncertain conditions and degrees of wear in remanufacturing, agile production systems that can adapt dynamically to changing requirements are needed. Machine learning algorithms can be employed due to their generalization capabilities of learning from various types and variants of products. However, in reality, datasets with a diversity of samples that can be used to train models are difficult to obtain in the initial period. This may cause bad performances when the system tries to adapt to new unseen input data in the future. In order to generate large datasets for different learning purposes, in our project, we present a Blender add-on named MotorFactory to generate customized mesh models of various motor instances. MotorFactory allows to create mesh models which, complemented with additional add-ons, can be further used to create synthetic RGB images, depth images, normal images, segmentation ground truth masks, and 3D point cloud datasets with point-wise semantic labels. The created synthetic datasets may be used for various tasks including motor type classification, object detection for decentralized material transfer tasks, part segmentation for disassembly and handling tasks, or even reinforcement learning-based robotics control or view-planning. △ Less

Submitted 11 January, 2023; originally announced January 2023.

SynMotor: A Benchmark Suite for Object Attribute Regression and Multi-task Learning

Authors: Chengzhi Wu, Linxi Qiu, Kanran Zhou, Julius Pfrommer, Jürgen Beyerer

Abstract: In this paper, we develop a novel benchmark suite including both a 2D synthetic image dataset and a 3D synthetic point cloud dataset. Our work is a sub-task in the framework of a remanufacturing project, in which small electric motors are used as fundamental objects. Apart from the given detection, classification, and segmentation annotations, the key objects also have multiple learnable attribute… ▽ More In this paper, we develop a novel benchmark suite including both a 2D synthetic image dataset and a 3D synthetic point cloud dataset. Our work is a sub-task in the framework of a remanufacturing project, in which small electric motors are used as fundamental objects. Apart from the given detection, classification, and segmentation annotations, the key objects also have multiple learnable attributes with ground truth provided. This benchmark can be used for computer vision tasks including 2D/3D detection, classification, segmentation, and multi-attribute learning. It is worth mentioning that most attributes of the motors are quantified as continuously variable rather than binary, which makes our benchmark well-suited for the less explored regression tasks. In addition, appropriate evaluation metrics are adopted or developed for each task and promising baseline results are provided. We hope this benchmark can stimulate more research efforts on the sub-domain of object attribute learning and multi-task learning in the future. △ Less

Submitted 11 January, 2023; originally announced January 2023.

Object Detection in 3D Point Clouds via Local Correlation-Aware Point Embedding

Authors: Chengzhi Wu, Julius Pfrommer, Jürgen Beyerer, Kangning Li, Boris Neubert

Abstract: We present an improved approach for 3D object detection in point cloud data based on the Frustum PointNet (F-PointNet). Compared to the original F-PointNet, our newly proposed method considers the point neighborhood when computing point features. The newly introduced local neighborhood embedding operation mimics the convolutional operations in 2D neural networks. Thus features of each point are no… ▽ More We present an improved approach for 3D object detection in point cloud data based on the Frustum PointNet (F-PointNet). Compared to the original F-PointNet, our newly proposed method considers the point neighborhood when computing point features. The newly introduced local neighborhood embedding operation mimics the convolutional operations in 2D neural networks. Thus features of each point are not only computed with the features of its own or of the whole point cloud but also computed especially with respect to the features of its neighbors. Experiments show that our proposed method achieves better performance than the F-Pointnet baseline on 3D object detection tasks. △ Less

Submitted 11 January, 2023; originally announced January 2023.

Self-Supervised Generative-Contrastive Learning of Multi-Modal Euclidean Input for 3D Shape Latent Representations: A Dynamic Switching Approach

Authors: Chengzhi Wu, Julius Pfrommer, Mingyuan Zhou, Jürgen Beyerer

Abstract: We propose a combined generative and contrastive neural architecture for learning latent representations of 3D volumetric shapes. The architecture uses two encoder branches for voxel grids and multi-view images from the same underlying shape. The main idea is to combine a contrastive loss between the resulting latent representations with an additional reconstruction loss. That helps to avoid colla… ▽ More We propose a combined generative and contrastive neural architecture for learning latent representations of 3D volumetric shapes. The architecture uses two encoder branches for voxel grids and multi-view images from the same underlying shape. The main idea is to combine a contrastive loss between the resulting latent representations with an additional reconstruction loss. That helps to avoid collapsing the latent representations as a trivial solution for minimizing the contrastive loss. A novel dynamic switching approach is used to cross-train two encoders with a shared decoder. The switching approach also enables the stop gradient operation on a random branch. Further classification experiments show that the latent representations learned with our self-supervised method integrate more useful information from the additional input data implicitly, thus leading to better reconstruction and classification performance. △ Less

Submitted 6 June, 2025; v1 submitted 11 January, 2023; originally announced January 2023.

Solving the unit-load pre-marshalling problem in block stacking storage systems with multiple access directions

Authors: Jakob Pfrommer, Anne Meyer, Kevin Tierney

Abstract: Block stacking storage systems are highly adaptable warehouse systems with low investment costs. With multiple, deep lanes they can achieve high storage densities, but accessing some unit loads can be time-consuming. The unit-load pre-marshalling problem sorts the unit loads in a block stacking storage system in off-peak time periods to prepare for upcoming orders. The goal is to find a minimum nu… ▽ More Block stacking storage systems are highly adaptable warehouse systems with low investment costs. With multiple, deep lanes they can achieve high storage densities, but accessing some unit loads can be time-consuming. The unit-load pre-marshalling problem sorts the unit loads in a block stacking storage system in off-peak time periods to prepare for upcoming orders. The goal is to find a minimum number of unit-load moves needed to sequence a storage bay in ascending order based on the retrieval priority group of each unit load. In this paper, we present two solution approaches for determining the minimum number of unit-load moves. We show that for storage bays with one access direction, it is possible to adapt existing, optimal tree search procedures and lower bound heuristics from the container pre-marshalling problem. For multiple access directions, we develop a novel, two-step solution approach based on a network flow model and an A* algorithm with an adapted lower bound that is applicable in all scenarios. We further analyze the performance of the presented solutions in computational experiments for randomly generated problem instances and show that multiple access directions greatly reduce both the total access time of unit loads and the required sorting effort. △ Less

Submitted 19 July, 2022; originally announced July 2022.

On the quest of low temperature nitrogen infusion relevant for superconducting Nb based radio-frequency cavities

Authors: G. D. L. Semione, A. D. Pandey, S. Tober, J. Pfrommer, A. Pouilan, J. Drnec, G. Schütz, T. F. Keller, H. Noei, V. Vonk, B. Foster, A. Stierle

Abstract: A detailed study of the near-surface structure and composition of Nb, the material of choice for Superconducting Radio Frequency accelerator (SRF) cavities, is of great importance in order to understand the effects of different treatments applied during cavity production. By means of surface-sensitive techniques such as grazing incidence diffuse X-ray scattering, X-ray reflectivity and X-ray photo… ▽ More A detailed study of the near-surface structure and composition of Nb, the material of choice for Superconducting Radio Frequency accelerator (SRF) cavities, is of great importance in order to understand the effects of different treatments applied during cavity production. By means of surface-sensitive techniques such as grazing incidence diffuse X-ray scattering, X-ray reflectivity and X-ray photoelectron spectroscopy, single-crystalline Nb(100) samples were investigated in and ex-situ during annealing in UHV as well as in nitrogen atmospheres with temperatures and pressures similar to the ones employed in real Nb cavity treatments. Annealing of Nb specimens up to 800°C in vacuum promotes partial reduction of the natural surface oxides (Nb2O5, NbO2, NbO) into NbO. Upon cooling to 120°C, no evidence of nitrogen-rich layers was detected after nitrogen exposure times of up to 48 hours. Oxygen enrichment below the Nb/oxide interface and posterior diffusion of oxygen species towards the Nb matrix, along with a partial reduction of the natural surface oxides was observed upon a stepwise annealing up to 250°C. Nitrogen introduction to the system at 250°C neither promotes N diffusion into the Nb matrix nor the formation of new surface layers. Upon further heating to 500°C in a nitrogen atmosphere, the growth of a new subsurface Nb$_x$N$_y$ layer was detected. These results shed light on the composition of the near-surface region of Nb after low-temperature nitrogen treatments, which are reported to lead to a performance enhancement of SRF cavities. △ Less

Submitted 27 June, 2019; originally announced June 2019.

Comments: 15 pages, 8 figures, 1 table, to be submitted to PRB

Journal ref: Phys. Rev. Accel. Beams 22, 103102 (2019)

Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems

Authors: Laura von Rueden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, Jannis Schuecker

Abstract: Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for inf… ▽ More Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed machine learning approaches. It considers the source of knowledge, its representation, and its integration into the machine learning pipeline. Based on this taxonomy, we survey related research and describe how different knowledge representations such as algebraic equations, logic rules, or simulation results can be used in learning systems. This evaluation of numerous papers on the basis of our taxonomy uncovers key methods in the field of informed machine learning. △ Less

Submitted 28 May, 2021; v1 submitted 29 March, 2019; originally announced March 2019.

Comments: Accepted at IEEE Transactions on Knowledge and Data Engineering: https://ieeexplore.ieee.org/document/9429985

Dynamic vehicle redistribution and online price incentives in shared mobility systems

Authors: Julius Pfrommer, Joseph Warrington, Georg Schildbach, Manfred Morari

Abstract: This paper considers a combination of intelligent repositioning decisions and dynamic pricing for the improved operation of shared mobility systems. The approach is applied to London's Barclays Cycle Hire scheme, which the authors have simulated based on historical data. Using model-based predictive control principles, dynamically varying rewards are computed and offered to customers carrying out… ▽ More This paper considers a combination of intelligent repositioning decisions and dynamic pricing for the improved operation of shared mobility systems. The approach is applied to London's Barclays Cycle Hire scheme, which the authors have simulated based on historical data. Using model-based predictive control principles, dynamically varying rewards are computed and offered to customers carrying out journeys. The aim is to encourage them to park bicycles at nearby under-used stations, thereby reducing the expected cost of repositioning them using dedicated staff. In parallel, the routes that repositioning staff should take are periodically recomputed using a model-based heuristic. It is shown that a trade-off between reward payouts to customers and the cost of hiring repositioning staff could be made, in order to minimize operating costs for a given desired service level. △ Less

Submitted 12 May, 2013; v1 submitted 14 April, 2013; originally announced April 2013.