CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Authors:
Hao Cui,
Zahra Shamsi,
Gowoon Cheon,
Xuejian Ma,
Shutong Li,
Maria Tikhanovskaya,
Peter Norgaard,
Nayantara Mudur,
Martyna Plomecka,
Paul Raccuglia,
Yasaman Bahri,
Victor V. Albert,
Pranesh Srinivasan,
Haining Pan,
Philippe Faist,
Brian Rohr,
Ekin Dogus Cubuk,
Muratahan Aykol,
Amil Merchant,
Michael J. Statt,
Dan Morris,
Drew Purves,
Elise Kleeman,
Ruth Alcantara,
Matthew Abraham
, et al. (9 additional authors not shown)
Abstract:
Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of…
▽ More
Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in https://github.com/google/curie
△ Less
Submitted 13 May, 2025; v1 submitted 14 March, 2025;
originally announced March 2025.
Dataset of Random Relaxations for Crystal Structure Search of Li-Si System
Authors:
Gowoon Cheon,
Lusann Yang,
Kevin McCloskey,
Evan J. Reed,
Ekin D. Cubuk
Abstract:
Crystal structure search is a long-standing challenge in materials design. We present a dataset of more than 100,000 structural relaxations of potential battery anode materials from randomized structures using density functional theory calculations. We illustrate the usage of the dataset by training graph neural networks to predict structural relaxations from randomly generated structures. Our mod…
▽ More
Crystal structure search is a long-standing challenge in materials design. We present a dataset of more than 100,000 structural relaxations of potential battery anode materials from randomized structures using density functional theory calculations. We illustrate the usage of the dataset by training graph neural networks to predict structural relaxations from randomly generated structures. Our models directly predict stresses in addition to forces, which allows them to accurately simulate relaxations of both ionic positions and lattice vectors. We show that models trained on the molecular dynamics simulations fail to simulate relaxations from random structures, while training on our data leads to up to two orders of magnitude decrease in error for the same task. Our model is able to find an experimentally verified structure of a stoichiometry held out from training. We find that randomly perturbing atomic positions during training improves both the accuracy and out of domain generalization of the models.
△ Less
Submitted 8 March, 2023; v1 submitted 4 December, 2020;
originally announced December 2020.
