-
Finance Language Model Evaluation (FLaME)
Authors:
Glenn Matlin,
Mika Okamoto,
Huzaifa Pardawala,
Yang Yang,
Sudheer Chava
Abstract:
Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs' performance on common Finance N…
▽ More
Language Models (LMs) have demonstrated impressive capabilities with core Natural Language Processing (NLP) tasks. The effectiveness of LMs for highly specialized knowledge-intensive tasks in finance remains difficult to assess due to major gaps in the methodologies of existing evaluation frameworks, which have caused an erroneous belief in a far lower bound of LMs' performance on common Finance NLP (FinNLP) tasks. To demonstrate the potential of LMs for these FinNLP tasks, we present the first holistic benchmarking suite for Financial Language Model Evaluation (FLaME). We are the first research paper to comprehensively study LMs against 'reasoning-reinforced' LMs, with an empirical study of 23 foundation LMs over 20 core NLP tasks in finance. We open-source our framework software along with all data and results.
△ Less
Submitted 18 June, 2025;
originally announced June 2025.
-
Construction and Analysis of Impression Caption Dataset for Environmental Sounds
Authors:
Yuki Okamoto,
Ryotaro Nagase,
Minami Okamoto,
Yuki Saito,
Keisuke Imoto,
Takahiro Fukumori,
Yoichi Yamashita
Abstract:
Some datasets with the described content and order of occurrence of sounds have been released for conversion between environmental sound and text. However, there are very few texts that include information on the impressions humans feel, such as "sharp" and "gorgeous," when they hear environmental sounds. In this study, we constructed a dataset with impression captions for environmental sounds tha…
▽ More
Some datasets with the described content and order of occurrence of sounds have been released for conversion between environmental sound and text. However, there are very few texts that include information on the impressions humans feel, such as "sharp" and "gorgeous," when they hear environmental sounds. In this study, we constructed a dataset with impression captions for environmental sounds that describe the impressions humans have when hearing these sounds. We used ChatGPT to generate impression captions and selected the most appropriate captions for sound by humans. Our dataset consists of 3,600 impression captions for environmental sounds. To evaluate the appropriateness of impression captions for environmental sounds, we conducted subjective and objective evaluations. From our evaluation results, we indicate that appropriate impression captions for environmental sounds can be generated.
△ Less
Submitted 20 October, 2024;
originally announced October 2024.
-
A GPU-Accelerated Moving-Horizon Algorithm for Training Deep Classification Trees on Large Datasets
Authors:
Jiayang Ren,
ValentÃn Osuna-Enciso,
Morimasa Okamoto,
Qiangqiang Mao,
Chaojie Ji,
Liang Cao,
Kaixun Hua,
Yankai Cao
Abstract:
Decision trees are essential yet NP-complete to train, prompting the widespread use of heuristic methods such as CART, which suffers from sub-optimal performance due to its greedy nature. Recently, breakthroughs in finding optimal decision trees have emerged; however, these methods still face significant computational costs and struggle with continuous features in large-scale datasets and deep tre…
▽ More
Decision trees are essential yet NP-complete to train, prompting the widespread use of heuristic methods such as CART, which suffers from sub-optimal performance due to its greedy nature. Recently, breakthroughs in finding optimal decision trees have emerged; however, these methods still face significant computational costs and struggle with continuous features in large-scale datasets and deep trees. To address these limitations, we introduce a moving-horizon differential evolution algorithm for classification trees with continuous features (MH-DEOCT). Our approach consists of a discrete tree decoding method that eliminates duplicated searches between adjacent samples, a GPU-accelerated implementation that significantly reduces running time, and a moving-horizon strategy that iteratively trains shallow subtrees at each node to balance the vision and optimizer capability. Comprehensive studies on 68 UCI datasets demonstrate that our approach outperforms the heuristic method CART on training and testing accuracy by an average of 3.44% and 1.71%, respectively. Moreover, these numerical studies empirically demonstrate that MH-DEOCT achieves near-optimal performance (only 0.38% and 0.06% worse than the global optimal method on training and testing, respectively), while it offers remarkable scalability for deep trees (e.g., depth=8) and large-scale datasets (e.g., ten million samples).
△ Less
Submitted 12 November, 2023;
originally announced November 2023.
-
Hibikino-Musashi@Home 2022 Team Description Paper
Authors:
Tomoya Shiba,
Tomohiro Ono,
Shoshi Tokuno,
Issei Uchino,
Masaya Okamoto,
Daiju Kanaoka,
Kazutaka Takahashi,
Kenta Tsukamoto,
Yoshiaki Tsutsumi,
Yugo Nakamura,
Yukiya Fukuda,
Yusuke Hoji,
Hayato Amano,
Yuma Kubota,
Mayu Koresawa,
Yoshifumi Sakai,
Ryogo Takemoto,
Katsunori Tamai,
Kazuo Nakahara,
Hiroyuki Hayashi,
Satsuki Fujimatsu,
Akinobu Mizutani,
Yusuke Mizoguchi,
Yuhei Yoshimitsu,
Mayo Suzuka
, et al. (5 additional authors not shown)
Abstract:
Our team, Hibikino-Musashi@Home (HMA), was founded in 2010. It is based in Japan in the Kitakyushu Science and Research Park. Since 2010, we have annually participated in the RoboCup@Home Japan Open competition in the open platform league (OPL).We participated as an open platform league team in the 2017 Nagoya RoboCup competition and as a domestic standard platform league (DSPL) team in the 2017 N…
▽ More
Our team, Hibikino-Musashi@Home (HMA), was founded in 2010. It is based in Japan in the Kitakyushu Science and Research Park. Since 2010, we have annually participated in the RoboCup@Home Japan Open competition in the open platform league (OPL).We participated as an open platform league team in the 2017 Nagoya RoboCup competition and as a domestic standard platform league (DSPL) team in the 2017 Nagoya, 2018 Montreal, 2019 Sydney, and 2021 Worldwide RoboCup competitions.We also participated in theWorld Robot Challenge (WRC) 2018 in the service-robotics category of the partner-robot challenge (real space) and won first place. Currently, we have 27 members from nine different laboratories within the Kyushu Institute of Technology and the university of Kitakyushu. In this paper, we introduce the activities that have been performed by our team and the technologies that we use.
△ Less
Submitted 12 November, 2022;
originally announced November 2022.
