-
Multi-Robot Task Planning for Multi-Object Retrieval Tasks with Distributed On-Site Knowledge via Large Language Models
Authors:
Kento Murata,
Shoichi Hasegawa,
Tomochika Ishikawa,
Yoshinobu Hagiwara,
Akira Taniguchi,
Lotfi El Hafi,
Tadahiro Taniguchi
Abstract:
It is crucial to efficiently execute instructions such as "Find an apple and a banana" or "Get ready for a field trip," which require searching for multiple objects or understanding context-dependent commands. This study addresses the challenging problem of determining which robot should be assigned to which part of a task when each robot possesses different situational on-site knowledge-specifica…
▽ More
It is crucial to efficiently execute instructions such as "Find an apple and a banana" or "Get ready for a field trip," which require searching for multiple objects or understanding context-dependent commands. This study addresses the challenging problem of determining which robot should be assigned to which part of a task when each robot possesses different situational on-site knowledge-specifically, spatial concepts learned from the area designated to it by the user. We propose a task planning framework that leverages large language models (LLMs) and spatial concepts to decompose natural language instructions into subtasks and allocate them to multiple robots. We designed a novel few-shot prompting strategy that enables LLMs to infer required objects from ambiguous commands and decompose them into appropriate subtasks. In our experiments, the proposed method achieved 47/50 successful assignments, outperforming random (28/50) and commonsense-based assignment (26/50). Furthermore, we conducted qualitative evaluations using two actual mobile manipulators. The results demonstrated that our framework could handle instructions, including those involving ad hoc categories such as "Get ready for a field trip," by successfully performing task decomposition, assignment, sequential planning, and execution.
△ Less
Submitted 30 September, 2025; v1 submitted 16 September, 2025;
originally announced September 2025.
-
Public Evaluation on Potential Social Impacts of Fully Autonomous Cybernetic Avatars for Physical Support in Daily-Life Environments: Large-Scale Demonstration and Survey at Avatar Land
Authors:
Lotfi El Hafi,
Kazuma Onishi,
Shoichi Hasegawa,
Akira Oyama,
Tomochika Ishikawa,
Masashi Osada,
Carl Tornberg,
Ryoma Kado,
Kento Murata,
Saki Hashimoto,
Sebastian Carrera Villalobos,
Akira Taniguchi,
Gustavo Alfonso Garcia Ricardez,
Yoshinobu Hagiwara,
Tatsuya Aoki,
Kensuke Iwata,
Takato Horii,
Yukiko Horikawa,
Takahiro Miyashita,
Tadahiro Taniguchi,
Hiroshi Ishiguro
Abstract:
Cybernetic avatars (CAs) are key components of an avatar-symbiotic society, enabling individuals to overcome physical limitations through virtual agents and robotic assistants. While semi-autonomous CAs intermittently require human teleoperation and supervision, the deployment of fully autonomous CAs remains a challenge. This study evaluates public perception and potential social impacts of fully…
▽ More
Cybernetic avatars (CAs) are key components of an avatar-symbiotic society, enabling individuals to overcome physical limitations through virtual agents and robotic assistants. While semi-autonomous CAs intermittently require human teleoperation and supervision, the deployment of fully autonomous CAs remains a challenge. This study evaluates public perception and potential social impacts of fully autonomous CAs for physical support in daily life. To this end, we conducted a large-scale demonstration and survey during Avatar Land, a 19-day public event in Osaka, Japan, where fully autonomous robotic CAs, alongside semi-autonomous CAs, performed daily object retrieval tasks. Specifically, we analyzed responses from 2,285 visitors who engaged with various CAs, including a subset of 333 participants who interacted with fully autonomous CAs and shared their perceptions and concerns through a survey questionnaire. The survey results indicate interest in CAs for physical support in daily life and at work. However, concerns were raised regarding task execution reliability. In contrast, cost and human-like interaction were not dominant concerns. Project page: https://lotfielhafi.github.io/FACA-Survey/.
△ Less
Submitted 16 July, 2025;
originally announced July 2025.
-
Development of a Practical Articulated Wheeled In-pipe Robot for Both 3-4 in Force Main Inspection of Sewer Pipes
Authors:
Kenya Murata,
Atsushi Kakogawa
Abstract:
This paper reports a practical articulated wheeled in-pipe inspection robot "AIRo-7.1" which is waterproof and dustproof, and can adapt to 3 to 4 in inner diameters. The joint torque can be adjusted by a PWM open-loop control. The middle joint angle can be controlled by a position feedback control system while the other two joints are bent by torsional springs. Thanks to this simple and high-densi…
▽ More
This paper reports a practical articulated wheeled in-pipe inspection robot "AIRo-7.1" which is waterproof and dustproof, and can adapt to 3 to 4 in inner diameters. The joint torque can be adjusted by a PWM open-loop control. The middle joint angle can be controlled by a position feedback control system while the other two joints are bent by torsional springs. Thanks to this simple and high-density design, not only downsizing of the robot but also wide range of the adaptive inner diameter were achieved. However, the relationship between the actual middle joint torque value and the PWM duty ratio should be pre-known because the reducer used in AIRo-7.1 was designed by ourselves. Therefore, preliminary experiments were conducted to clarify the relationship between them. To examine the adaptive movement, experiments in both 3 in and 4 in pipes with vertical, bend, and diameter change sections. Finally, field experiment was also conducted. From the results, high adaptability to different inner diameters of pipes and slippery environments were confirmed although waterproof and dustproof were not perfectly working.
△ Less
Submitted 5 November, 2024;
originally announced November 2024.
-
Visual-based Positioning and Pose Estimation
Authors:
Somnuk Phon-Amnuaisuk,
Ken T. Murata,
La-Or Kovavisaruch,
Tiong-Hoo Lim,
Praphan Pavarangkoon,
Takamichi Mizuhara
Abstract:
Recent advances in deep learning and computer vision offer an excellent opportunity to investigate high-level visual analysis tasks such as human localization and human pose estimation. Although the performance of human localization and human pose estimation has significantly improved in recent reports, they are not perfect and erroneous localization and pose estimation can be expected among video…
▽ More
Recent advances in deep learning and computer vision offer an excellent opportunity to investigate high-level visual analysis tasks such as human localization and human pose estimation. Although the performance of human localization and human pose estimation has significantly improved in recent reports, they are not perfect and erroneous localization and pose estimation can be expected among video frames. Studies on the integration of these techniques into a generic pipeline that is robust to noise introduced from those errors are still lacking. This paper fills the missing study. We explored and developed two working pipelines that suited the visual-based positioning and pose estimation tasks. Analyses of the proposed pipelines were conducted on a badminton game. We showed that the concept of tracking by detection could work well, and errors in position and pose could be effectively handled by a linear interpolation technique using information from nearby frames. The results showed that the Visual-based Positioning and Pose Estimation could deliver position and pose estimations with good spatial and temporal resolutions.
△ Less
Submitted 20 April, 2022;
originally announced April 2022.
-
Exploring the Applications of Faster R-CNN and Single-Shot Multi-box Detection in a Smart Nursery Domain
Authors:
Somnuk Phon-Amnuaisuk,
Ken T. Murata,
Praphan Pavarangkoon,
Kazunori Yamamoto,
Takamichi Mizuhara
Abstract:
The ultimate goal of a baby detection task concerns detecting the presence of a baby and other objects in a sequence of 2D images, tracking them and understanding the semantic contents of the scene. Recent advances in deep learning and computer vision offer various powerful tools in general object detection and can be applied to a baby detection task. In this paper, the Faster Region-based Convolu…
▽ More
The ultimate goal of a baby detection task concerns detecting the presence of a baby and other objects in a sequence of 2D images, tracking them and understanding the semantic contents of the scene. Recent advances in deep learning and computer vision offer various powerful tools in general object detection and can be applied to a baby detection task. In this paper, the Faster Region-based Convolutional Neural Network and the Single-Shot Multi-Box Detection approaches are explored. They are the two state-of-the-art object detectors based on the region proposal tactic and the multi-box tactic. The presence of a baby in the scene obtained from these detectors, tested using different pre-trained models, are discussed. This study is important since the behaviors of these detectors in a baby detection task using different pre-trained models are still not well understood. This exploratory study reveals many useful insights into the applications of these object detectors in the smart nursery domain.
△ Less
Submitted 26 August, 2018;
originally announced August 2018.
