-
J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM
Authors:
Takero Yoshida,
Yuikazu Ito,
Yoshihiro Fujiwara,
Shinji Tsuchida,
Daisuke Sugiyama,
Daisuke Matsuoka
Abstract:
Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of mari…
▽ More
Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.
△ Less
Submitted 20 December, 2024;
originally announced December 2024.
-
Automatic Dance Video Segmentation for Understanding Choreography
Authors:
Koki Endo,
Shuhei Tsuchida,
Tsukasa Fukusato,
Takeo Igarashi
Abstract:
Segmenting dance video into short movements is a popular way to easily understand dance choreography. However, it is currently done manually and requires a significant amount of effort by experts. That is, even if many dance videos are available on social media (e.g., TikTok and YouTube), it remains difficult for people, especially novices, to casually watch short video segments to practice dance…
▽ More
Segmenting dance video into short movements is a popular way to easily understand dance choreography. However, it is currently done manually and requires a significant amount of effort by experts. That is, even if many dance videos are available on social media (e.g., TikTok and YouTube), it remains difficult for people, especially novices, to casually watch short video segments to practice dance choreography. In this paper, we propose a method to automatically segment a dance video into each movement. Given a dance video as input, we first extract visual and audio features: the former is computed from the keypoints of the dancer in the video, and the latter is computed from the Mel spectrogram of the music in the video. Next, these features are passed to a Temporal Convolutional Network (TCN), and segmentation points are estimated by picking peaks of the network output. To build our training dataset, we annotate segmentation points to dance videos in the AIST Dance Video Database, which is a shared database containing original street dance videos with copyright-cleared dance music. The evaluation study shows that the proposed method (i.e., combining the visual and audio features) can estimate segmentation points with high accuracy. In addition, we developed an application to help dancers practice choreography using the proposed method.
△ Less
Submitted 30 May, 2024;
originally announced May 2024.
-
Embedded System Evolution in IoT System Development Based on MAPE-K Loop Mechanism
Authors:
Hiroyuki Nakagawa,
Shinya Tsuchida,
Emiliano Tramontana,
Andrea Fornaia,
Tatsuhiro Tsuchiya
Abstract:
Embedded systems including IoT devices are designed for specialized functions; thus, changes in functions are not considered following their release. For this reason, changing functions to satisfy the requirements of IoT systems is difficult. In this study, we focus on updating existing embedded systems without modifying them. We investigate the design of new functions and their implementation wit…
▽ More
Embedded systems including IoT devices are designed for specialized functions; thus, changes in functions are not considered following their release. For this reason, changing functions to satisfy the requirements of IoT systems is difficult. In this study, we focus on updating existing embedded systems without modifying them. We investigate the design of new functions and their implementation with limited resources. This paper describes an evolution mechanism for updating the functionalities of existing embedded systems. The evolution mechanism uses a control unit that is deployed outside the embedded system. To guide the steady implementation of the evolution mechanism, we define an evolution process that effectively uses the state machine diagram at the design time and runtime to update the embedded systems. The programming framework implemented in this study supports the evolution process. We evaluate the evolution mechanism based on the results from two experiments. The first experiment involved applying the evolution mechanism to a cleaning robot, this demonstrated that the evolution mechanism systematically enables the injection of new functions into an embedded system in the real world. The second experiment, on the probabilistic model checking, demonstrated that the mechanism provides almost the same performance as the ordinary embedded system with an improved robustness.
△ Less
Submitted 26 May, 2022;
originally announced May 2022.
