-
Open FinLLM Leaderboard: Towards Financial AI Readiness
Authors:
Shengyuan Colin Lin,
Felix Tian,
Keyi Wang,
Xingjian Zhao,
Jimin Huang,
Qianqian Xie,
Luca Borella,
Matt White,
Christina Dan Wang,
Kairong Xiao,
Xiao-Yang Liu Yanglet,
Li Deng
Abstract:
Financial large language models (FinLLMs) with multimodal capabilities are envisioned to revolutionize applications across business, finance, accounting, and auditing. However, real-world adoption requires robust benchmarks of FinLLMs' and FinAgents' performance. Maintaining an open leaderboard is crucial for encouraging innovative adoption and improving model effectiveness. In collaboration with…
▽ More
Financial large language models (FinLLMs) with multimodal capabilities are envisioned to revolutionize applications across business, finance, accounting, and auditing. However, real-world adoption requires robust benchmarks of FinLLMs' and FinAgents' performance. Maintaining an open leaderboard is crucial for encouraging innovative adoption and improving model effectiveness. In collaboration with Linux Foundation and Hugging Face, we create an open FinLLM leaderboard, which serves as an open platform for assessing and comparing AI models' performance on a wide spectrum of financial tasks. By demoncratizing access to advances of financial knowledge and intelligence, a chatbot or agent may enhance the analytical capabilities of the general public to a professional level within a few months of usage. This open leaderboard welcomes contributions from academia, open-source community, industry, and stakeholders. In particular, we encourage contributions of new datasets, tasks, and models for continual update. Through fostering a collaborative and open ecosystem, we seek to promote financial AI readiness.
△ Less
Submitted 29 April, 2025; v1 submitted 19 January, 2025;
originally announced January 2025.
-
A Report on Financial Regulations Challenge at COLING 2025
Authors:
Keyi Wang,
Jaisal Patel,
Charlie Shen,
Daniel Kim,
Andy Zhu,
Alex Lin,
Luca Borella,
Cailean Osborne,
Matt White,
Steve Yang,
Kairong Xiao,
Xiao-Yang Liu Yanglet
Abstract:
Financial large language models (FinLLMs) have been applied to various tasks in business, finance, accounting, and auditing. Complex financial regulations and standards are critical to financial services, which LLMs must comply with. However, FinLLMs' performance in understanding and interpreting financial regulations has rarely been studied. Therefore, we organize the Regulations Challenge, a sha…
▽ More
Financial large language models (FinLLMs) have been applied to various tasks in business, finance, accounting, and auditing. Complex financial regulations and standards are critical to financial services, which LLMs must comply with. However, FinLLMs' performance in understanding and interpreting financial regulations has rarely been studied. Therefore, we organize the Regulations Challenge, a shared task at COLING 2025. It encourages the academic community to explore the strengths and limitations of popular LLMs. We create 9 novel tasks and corresponding question sets. In this paper, we provide an overview of these tasks and summarize participants' approaches and results. We aim to raise awareness of FinLLMs' professional capability in financial regulations.
△ Less
Submitted 12 January, 2025; v1 submitted 15 December, 2024;
originally announced December 2024.
-
Ultra-low latency quantum-inspired machine learning predictors implemented on FPGA
Authors:
Lorenzo Borella,
Alberto Coppi,
Jacopo Pazzini,
Andrea Stanco,
Marco Trenti,
Andrea Triossi,
Marco Zanetti
Abstract:
Tensor Networks (TNs) are a computational paradigm used for representing quantum many-body systems. Recent works have shown how TNs can also be applied to perform Machine Learning (ML) tasks, yielding comparable results to standard supervised learning techniques. In this work, we study the use of Tree Tensor Networks (TTNs) in high-frequency real-time applications by exploiting the low-latency har…
▽ More
Tensor Networks (TNs) are a computational paradigm used for representing quantum many-body systems. Recent works have shown how TNs can also be applied to perform Machine Learning (ML) tasks, yielding comparable results to standard supervised learning techniques. In this work, we study the use of Tree Tensor Networks (TTNs) in high-frequency real-time applications by exploiting the low-latency hardware of the Field-Programmable Gate Array (FPGA) technology. We present different implementations of TTN classifiers, capable of performing inference on classical ML datasets as well as on complex physics data. A preparatory analysis of bond dimensions and weight quantization is realized in the training phase, together with entanglement entropy and correlation measurements, that help setting the choice of the TTN architecture. The generated TTNs are then deployed on a hardware accelerator; using an FPGA integrated into a server, the inference of the TTN is completely offloaded. Eventually, a classifier for High Energy Physics (HEP) applications is implemented and executed fully pipelined with sub-microsecond latency.
△ Less
Submitted 25 September, 2024; v1 submitted 24 September, 2024;
originally announced September 2024.
