-
The Surprising Soupability of Documents in State Space Models
Authors:
Yasaman Jafari,
Zixian Wang,
Leon Bergen,
Taylor Berg-Kirkpatrick
Abstract:
We investigate whether hidden states from Structured State Space Models (SSMs) can be merged post-hoc to support downstream reasoning. Inspired by model souping, we propose a strategy where documents are encoded independently and their representations are pooled -- via simple operations like averaging -- into a single context state. This approach, which we call document souping, enables modular en…
▽ More
We investigate whether hidden states from Structured State Space Models (SSMs) can be merged post-hoc to support downstream reasoning. Inspired by model souping, we propose a strategy where documents are encoded independently and their representations are pooled -- via simple operations like averaging -- into a single context state. This approach, which we call document souping, enables modular encoding and reuse without reprocessing the full input for each query. We finetune Mamba2 models to produce soupable representations and find that they support multi-hop QA, sparse retrieval, and long-document reasoning with strong accuracy. On HotpotQA, souping ten independently encoded documents nearly matches the performance of a cross-encoder trained on the same inputs.
△ Less
Submitted 29 May, 2025;
originally announced May 2025.
-
ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models
Authors:
Veeramakali Vignesh Manivannan,
Yasaman Jafari,
Srikar Eranky,
Spencer Ho,
Rose Yu,
Duncan Watson-Parris,
Yian Ma,
Leon Bergen,
Taylor Berg-Kirkpatrick
Abstract:
The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from…
▽ More
The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa
△ Less
Submitted 9 March, 2025; v1 submitted 22 October, 2024;
originally announced October 2024.
-
MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimization
Authors:
Yasaman Jafari,
Dheeraj Mekala,
Rose Yu,
Taylor Berg-Kirkpatrick
Abstract:
RL-based techniques can be employed to search for prompts that, when fed into a target language model, maximize a set of user-specified reward functions. However, in many target applications, the natural reward functions are in tension with one another -- for example, content preservation vs. style matching in style transfer tasks. Current techniques focus on maximizing the average of reward funct…
▽ More
RL-based techniques can be employed to search for prompts that, when fed into a target language model, maximize a set of user-specified reward functions. However, in many target applications, the natural reward functions are in tension with one another -- for example, content preservation vs. style matching in style transfer tasks. Current techniques focus on maximizing the average of reward functions, which does not necessarily lead to prompts that achieve balance across rewards -- an issue that has been well-studied in the multi-objective and robust optimization literature. In this paper, we conduct an empirical comparison of several existing multi-objective optimization techniques adapted to this new setting: RL-based discrete prompt optimization. We compare two methods optimizing the volume of the Pareto reward surface and one method that chooses an update direction that benefits all rewards simultaneously. We evaluate performance on two NLP tasks: style transfer and machine translation, each using three competing reward functions. Our experiments demonstrate that multi-objective methods that directly optimize the volume of the Pareto reward surface perform better and achieve a better balance of all rewards than those that attempt to find monotonic update directions.
△ Less
Submitted 16 October, 2024; v1 submitted 18 February, 2024;
originally announced February 2024.
-
Investigating the effects of Goodreads challenges on individuals reading habits
Authors:
Yasaman Jafari,
Nazanin Sabri,
Behnam Bahrak
Abstract:
Sharing our goals with others and setting public challenges for ourselves is a topic that has been the center of many discussions. This study examines reading challenges, how participation in them has changed throughout the years, and how they influence users reading productivity. To do so, we analyze Goodreads, a social book cataloging website, with a yearly challenge feature. We further show tha…
▽ More
Sharing our goals with others and setting public challenges for ourselves is a topic that has been the center of many discussions. This study examines reading challenges, how participation in them has changed throughout the years, and how they influence users reading productivity. To do so, we analyze Goodreads, a social book cataloging website, with a yearly challenge feature. We further show that gender is a significant factor in how successful individuals are in their challenges. Additionally, we investigate the association between participation in reading challenges and the number of books people read.
△ Less
Submitted 15 February, 2021; v1 submitted 7 December, 2020;
originally announced December 2020.
