-
LLM-based Agent Simulation for Maternal Health Interventions: Uncertainty Estimation and Decision-focused Evaluation
Authors:
Sarah Martinson,
Lingkai Kong,
Cheol Woo Kim,
Aparna Taneja,
Milind Tambe
Abstract:
Agent-based simulation is crucial for modeling complex human behavior, yet traditional approaches require extensive domain knowledge and large datasets. In data-scarce healthcare settings where historic and counterfactual data are limited, large language models (LLMs) offer a promising alternative by leveraging broad world knowledge. This study examines an LLM-driven simulation of a maternal mobil…
▽ More
Agent-based simulation is crucial for modeling complex human behavior, yet traditional approaches require extensive domain knowledge and large datasets. In data-scarce healthcare settings where historic and counterfactual data are limited, large language models (LLMs) offer a promising alternative by leveraging broad world knowledge. This study examines an LLM-driven simulation of a maternal mobile health program, predicting beneficiaries' listening behavior when they receive health information via automated messages (control) or live representatives (intervention). Since uncertainty quantification is critical for decision-making in health interventions, we propose an LLM epistemic uncertainty estimation method based on binary entropy across multiple samples. We enhance model robustness through ensemble approaches, improving F1 score and model calibration compared to individual models. Beyond direct evaluation, we take a decision-focused approach, demonstrating how LLM predictions inform intervention feasibility and trial implementation in data-limited settings. The proposed method extends to public health, disaster response, and other domains requiring rapid intervention assessment under severe data constraints. All code and prompts used for this work can be found at https://github.com/sarahmart/LLM-ABS-ARMMAN-prediction.
△ Less
Submitted 25 March, 2025;
originally announced March 2025.
-
Humanity's Last Exam
Authors:
Long Phan,
Alice Gatti,
Ziwen Han,
Nathaniel Li,
Josephina Hu,
Hugh Zhang,
Chen Bo Calvin Zhang,
Mohamed Shaaban,
John Ling,
Sean Shi,
Michael Choi,
Anish Agrawal,
Arnav Chopra,
Adam Khoja,
Ryan Kim,
Richard Ren,
Jason Hausenloy,
Oliver Zhang,
Mantas Mazeika,
Dmitry Dodonov,
Tung Nguyen,
Jaeho Lee,
Daron Anderson,
Mikhail Doroshenko,
Alun Cennyth Stokes
, et al. (1084 additional authors not shown)
Abstract:
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of…
▽ More
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
△ Less
Submitted 19 April, 2025; v1 submitted 24 January, 2025;
originally announced January 2025.
-
HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
Authors:
Jingxuan Fan,
Sarah Martinson,
Erik Y. Wang,
Kaylie Hausknecht,
Jonah Brenner,
Danxian Liu,
Nianli Peng,
Corey Wang,
Michael P. Brenner
Abstract:
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational t…
▽ More
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMath, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like HARDMath to advance mathematical abilities of LLMs.
△ Less
Submitted 13 December, 2024; v1 submitted 13 October, 2024;
originally announced October 2024.
-
Decay Energy Spectrometry for Improved Nuclear Material Analysis at the IAEA NML
Authors:
G. B. Kim,
A. R. L. Kavner,
T. Parsons-Davis,
S. Friedrich,
O. B. Drury,
D. Lee,
X. Zhang,
N. Hines,
S. T. P. Boyd,
S. Weidenbenner,
K. Schreiber,
S. Martinson,
C. Smith,
D. McNeel,
S. Salazar,
K. Koehler,
M. Carpenter,
M. Croce,
D. Schmidt,
J. Ullom
Abstract:
Decay energy spectrometry (DES) is a novel radiometric technique for high-precision analysis of nuclear materials. DES employs the unique thermal detection physics of cryogenic microcalorimeters with ultra-high energy resolution and 100$\%$ detection efficiency to accomplish high precision decay energy measurements. Low-activity nuclear samples of 1 Bq or less, and without chemical separation, are…
▽ More
Decay energy spectrometry (DES) is a novel radiometric technique for high-precision analysis of nuclear materials. DES employs the unique thermal detection physics of cryogenic microcalorimeters with ultra-high energy resolution and 100$\%$ detection efficiency to accomplish high precision decay energy measurements. Low-activity nuclear samples of 1 Bq or less, and without chemical separation, are used to provide elemental and isotopic compositions in a single measurement. Isotopic ratio precisions of 1 ppm - 1,000 ppm (isotope dependent), which is close to that of the mass spectrometry, have been demonstrated in 12-hour DES measurements of ~5 Bq samples of certified reference materials of uranium (U) and plutonium (Pu). DES has very different systematic biases and uncertainties, as well as different sensitivities to nuclides, compared to mass-spectrometry techniques. Therefore, the accuracy and confidence of nuclear material assays can be improved by combining this new technique with existing mass-spectrometry techniques. Commercial-level DES techniques and equipment are being developed for the implementation of DES at the Nuclear Material Laboratory (NML) of International Atomic Energy Agency (IAEA) to provide complementary measurements to the existing technologies. The paper describes details of DES measurement methods, as well as DES precision and accuracy to U and Pu standard sources to discuss its capability in analysis of nuclear safeguards samples.
△ Less
Submitted 11 July, 2024; v1 submitted 7 June, 2024;
originally announced June 2024.
