-
Smoothie: Label Free Language Model Routing
Authors:
Neel Guha,
Mayee F. Chen,
Trevor Chow,
Ishan S. Khare,
Christopher Ré
Abstract:
Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require trainin…
▽ More
Large language models (LLMs) are increasingly used in applications where LLM inputs may span many different tasks. Recent work has found that the choice of LLM is consequential, and different LLMs may be good for different input samples. Prior approaches have thus explored how engineers might select an LLM to use for each sample (i.e. routing). While existing routing methods mostly require training auxiliary models on human-annotated data, our work explores whether it is possible to perform unsupervised routing. We propose Smoothie, a weak supervision-inspired routing approach that requires no labeled data. Given a set of outputs from different LLMs, Smoothie constructs a latent variable graphical model over embedding representations of observable LLM outputs and unknown "true" outputs. Using this graphical model, we estimate sample-dependent quality scores for each LLM, and route each sample to the LLM with the highest corresponding score. We find that Smoothie's LLM quality-scores correlate with ground-truth model quality (correctly identifying the optimal model on 9/14 tasks), and that Smoothie outperforms baselines for routing by up to 10 points accuracy.
△ Less
Submitted 5 December, 2024;
originally announced December 2024.
-
WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks
Authors:
Michael Wornow,
Avanika Narayan,
Ben Viggiano,
Ishan S. Khare,
Tathagat Verma,
Tibor Thompson,
Miguel Angel Fuentes Hernandez,
Sudharsan Sundar,
Chloe Trujillo,
Krrish Chawla,
Rongfei Lu,
Justin Shen,
Divya Nagaraj,
Joshua Martinez,
Vardhan Agrawal,
Althea Hudson,
Nigam H. Shah,
Christopher Re
Abstract:
Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This f…
▽ More
Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This focus on automation ignores the reality of how most BPM tools are applied today - simply documenting the relevant workflow takes 60% of the time of the typical process optimization project. To address this gap we present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM tasks beyond automation. Our contributions are: (1) a dataset containing 2928 documented workflow demonstrations; (2) 6 novel BPM tasks sourced from real-world applications ranging from workflow documentation to knowledge transfer to process improvement; and (3) an automated evaluation harness. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation (e.g. recalling 88% of the steps taken in a video demonstration of a workflow), they struggle to re-apply that knowledge towards finer-grained validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages the development of more "human-centered" AI tooling for enterprise applications and furthers the exploration of multimodal FMs for the broader universe of BPM tasks. We publish our dataset and experiments here: https://github.com/HazyResearch/wonderbread
△ Less
Submitted 10 October, 2024; v1 submitted 19 June, 2024;
originally announced June 2024.
-
Evaluation of Reinforcement Learning Techniques for Trading on a Diverse Portfolio
Authors:
Ishan S. Khare,
Tarun K. Martheswaran,
Akshana Dassanaike-Perera
Abstract:
This work seeks to answer key research questions regarding the viability of reinforcement learning over the S&P 500 index. The on-policy techniques of Value Iteration (VI) and State-action-reward-state-action (SARSA) are implemented along with the off-policy technique of Q-Learning. The models are trained and tested on a dataset comprising multiple years of stock market data from 2000-2023. The an…
▽ More
This work seeks to answer key research questions regarding the viability of reinforcement learning over the S&P 500 index. The on-policy techniques of Value Iteration (VI) and State-action-reward-state-action (SARSA) are implemented along with the off-policy technique of Q-Learning. The models are trained and tested on a dataset comprising multiple years of stock market data from 2000-2023. The analysis presents the results and findings from training and testing the models using two different time periods: one including the COVID-19 pandemic years and one excluding them. The results indicate that including market data from the COVID-19 period in the training dataset leads to superior performance compared to the baseline strategies. During testing, the on-policy approaches (VI and SARSA) outperform Q-learning, highlighting the influence of bias-variance tradeoff and the generalization capabilities of simpler policies. However, it is noted that the performance of Q-learning may vary depending on the stability of future market conditions. Future work is suggested, including experiments with updated Q-learning policies during testing and trading diverse individual stocks. Additionally, the exploration of alternative economic indicators for training the models is proposed.
△ Less
Submitted 10 February, 2024; v1 submitted 28 June, 2023;
originally announced September 2023.
-
Actionable Insights on Philadelphia Crime Hot-Spots: Clustering and Statistical Analysis to Inform Future Crime Legislation
Authors:
Ishan S. Khare,
Tarun K. Martheswaran,
Rahul K. Thomas,
Aditya Bora
Abstract:
Philadelphia's problem with high crime rates continues to be exacerbated as Philadelphia's residents, community leaders, and law enforcement officials struggle to address the root causes of the problem and make the city safer for all. In this work, we deeply understand crime in Philadelphia and offer novel insights for crime mitigation within the city. Open source crime data from 2012-2022 was obt…
▽ More
Philadelphia's problem with high crime rates continues to be exacerbated as Philadelphia's residents, community leaders, and law enforcement officials struggle to address the root causes of the problem and make the city safer for all. In this work, we deeply understand crime in Philadelphia and offer novel insights for crime mitigation within the city. Open source crime data from 2012-2022 was obtained from OpenDataPhilly. Density-Based Spatial Clustering of Applications with Noise (DBSCAN) was used to cluster geographic locations of crimes. Clustering of crimes within each of 21 police districts was performed, and temporal changes in cluster distributions were analyzed to develop a Non-Systemic Index (NSI). Home Owners' Loan Corporation (HOLC) grades were tested for associations with clusters in police districts labeled `systemic.' Crimes within each district were highly clusterable, according to Hopkins' Mean Statistics. NSI proved to be a good measure of differentiating systemic ($<$ 0.06) and non-systemic ($\geq$ 0.06) districts. Two systemic districts, 19 and 25, were found to be significantly correlated with HOLC grade (p $=2.02 \times 10^{-19}$, p $=1.52 \times 10^{-13}$). Philadelphia crime data shows a high level of heterogeneity between districts. Classification of districts with NSI allows for targeted crime mitigation strategies. Policymakers can interpret this work as a guide to interventions.
△ Less
Submitted 28 June, 2023;
originally announced June 2023.
-
Electronic, optical and thermoelectric properties of sodium pnictogen chalcogenides: A first principles study
Authors:
I. S. Khare,
N. J. Szymanski,
D. Gall,
R. E. Irving
Abstract:
Ternary chalcogenides have been of recent investigation for applications in photovoltaic and thermoelectric devices. We study the structural, electronic, optical, and thermoelectric properties of nine ternary chalcogenides, NaAX$_{2}$, where A represents pnictogens (As, Sb, and Te) and X represents chalcogens (S, Se, and Te). Calculations based on density functional theory yield the following resu…
▽ More
Ternary chalcogenides have been of recent investigation for applications in photovoltaic and thermoelectric devices. We study the structural, electronic, optical, and thermoelectric properties of nine ternary chalcogenides, NaAX$_{2}$, where A represents pnictogens (As, Sb, and Te) and X represents chalcogens (S, Se, and Te). Calculations based on density functional theory yield the following results: (i) phonon dispersion curves predict three of the compounds, NaAsS$_{2}$, NaSbS$_{2}$, and NaSbSe$_{2}$, to be dynamically stable in the monoclinic, C2/c, structure, (ii) the layered atomistic configuration causes the corresponding electronic and optical properties to display a high degree of anisotropy, (iii) A-X electronic bonding features vary significantly with structural distortions arising from atomic size mismatch, therefore directly influencing stability, (iv) strong absorption is observed in the stable compounds, with coefficients ranging from 10$^{4}$ to 10$^{5}$ cm$^{-1}$ in the visible-UV range, and (v) remarkably high Seebeck coefficients exceeding 500 $μ$V/K at carrier concentrations commonly achieved in such materials. From these results, we conclude that NaAsS$_{2}$, NaSbS$_{2}$, and NaSbSe$_{2}$ are suitable candidates for both photovoltaic, particularly in tandem solar cells, and thermoelectric applications. Experimental synthesis and verification are suggested.
△ Less
Submitted 3 January, 2020;
originally announced January 2020.
