-
Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Tracking
Authors:
Christopher Richardson,
Roshan Sharma,
Neeraj Gaur,
Parisa Haghani,
Anirudh Sundar,
Bhuvana Ramabhadran
Abstract:
Zero-shot domain adaptation for dialogue state tracking (DST) remains a challenging problem in task-oriented dialogue (TOD) systems, where models must generalize to target domains unseen at training time. Current large language model approaches for zero-shot domain adaptation rely on prompting to introduce knowledge pertaining to the target domains. However, their efficacy strongly depends on prom…
▽ More
Zero-shot domain adaptation for dialogue state tracking (DST) remains a challenging problem in task-oriented dialogue (TOD) systems, where models must generalize to target domains unseen at training time. Current large language model approaches for zero-shot domain adaptation rely on prompting to introduce knowledge pertaining to the target domains. However, their efficacy strongly depends on prompt engineering, as well as the zero-shot ability of the underlying language model. In this work, we devise a novel data augmentation approach, Schema Augmentation, that improves the zero-shot domain adaptation of language models through fine-tuning. Schema Augmentation is a simple but effective technique that enhances generalization by introducing variations of slot names within the schema provided in the prompt. Experiments on MultiWOZ and SpokenWOZ showed that the proposed approach resulted in a substantial improvement over the baseline, in some experiments achieving over a twofold accuracy gain over unseen domains while maintaining equal or superior performance over all domains.
△ Less
Submitted 21 February, 2025; v1 submitted 31 October, 2024;
originally announced November 2024.
-
cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers
Authors:
Anirudh Sundar,
Jin Xu,
William Gay,
Christopher Richardson,
Larry Heck
Abstract:
An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed of text, equations, figures, and tables, SIMMC methods must be developed specifically for each component to support the depth of inquiry and interactions required by research scientists. This work introduces Conversatio…
▽ More
An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed of text, equations, figures, and tables, SIMMC methods must be developed specifically for each component to support the depth of inquiry and interactions required by research scientists. This work introduces Conversational Papers (cPAPERS), a dataset of conversational question-answer pairs from reviews of academic papers grounded in these paper components and their associated references from scientific documents available on arXiv. We present a data collection strategy to collect these question-answer pairs from OpenReview and associate them with contextual information from LaTeX source files. Additionally, we present a series of baseline approaches utilizing Large Language Models (LLMs) in both zero-shot and fine-tuned configurations to address the cPAPERS dataset.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
iTBLS: A Dataset of Interactive Conversations Over Tabular Information
Authors:
Anirudh Sundar,
Christopher Richardson,
William Gay,
Larry Heck
Abstract:
This paper introduces Interactive Tables (iTBLS), a dataset of interactive conversations situated in tables from scientific articles. This dataset is designed to facilitate human-AI collaborative problem-solving through AI-powered multi-task tabular capabilities. In contrast to prior work that models interactions as factoid QA or procedure synthesis, iTBLS broadens the scope of interactions to inc…
▽ More
This paper introduces Interactive Tables (iTBLS), a dataset of interactive conversations situated in tables from scientific articles. This dataset is designed to facilitate human-AI collaborative problem-solving through AI-powered multi-task tabular capabilities. In contrast to prior work that models interactions as factoid QA or procedure synthesis, iTBLS broadens the scope of interactions to include mathematical reasoning, natural language manipulation, and expansion of existing tables from natural language conversation by delineating interactions into one of three tasks: interpretation, modification, or generation. Additionally, the paper presents a suite of baseline approaches to iTBLS, utilizing zero-shot prompting and parameter-efficient fine-tuning for different computing situations. We also introduce a novel multi-step approach and show how it can be leveraged in conjunction with parameter-efficient fine-tuning to achieve the state-of-the-art on iTBLS; outperforming standard parameter-efficient fine-tuning by up to 15% on interpretation, 18% on modification, and 38% on generation.
△ Less
Submitted 18 April, 2024;
originally announced April 2024.
-
gTBLS: Generating Tables from Text by Conditional Question Answering
Authors:
Anirudh Sundar,
Christopher Richardson,
Larry Heck
Abstract:
Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables is ensuring their syntactic validity. Prior approaches address this challenge by including additional parameters in the Transformer's attention mechanism to attend to specific rows and column headers. In contrast to this…
▽ More
Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables is ensuring their syntactic validity. Prior approaches address this challenge by including additional parameters in the Transformer's attention mechanism to attend to specific rows and column headers. In contrast to this single-stage method, this paper presents a two-stage approach called Generative Tables (gTBLS). The first stage infers table structure (row and column headers) from the text. The second stage formulates questions using these headers and fine-tunes a causal language model to answer them. Furthermore, the gTBLS approach is amenable to the utilization of pre-trained Large Language Models in a zero-shot configuration, presenting a solution for table generation in situations where fine-tuning is not feasible. gTBLS improves prior approaches by up to 10% in BERTScore on the table construction task and up to 20% on the table content generation task of the E2E, WikiTableText, WikiBio, and RotoWire datasets.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models
Authors:
Chris Richardson,
Yao Zhang,
Kellen Gillespie,
Sudipta Kar,
Arshdeep Singh,
Zeynab Raeesy,
Omar Zia Khan,
Abhinav Sethy
Abstract:
Personalization, the ability to tailor a system to individual users, is an essential factor in user experience with natural language processing (NLP) systems. With the emergence of Large Language Models (LLMs), a key question is how to leverage these models to better personalize user experiences. To personalize a language model's output, a straightforward approach is to incorporate past user data…
▽ More
Personalization, the ability to tailor a system to individual users, is an essential factor in user experience with natural language processing (NLP) systems. With the emergence of Large Language Models (LLMs), a key question is how to leverage these models to better personalize user experiences. To personalize a language model's output, a straightforward approach is to incorporate past user data into the language model prompt, but this approach can result in lengthy inputs exceeding limitations on input length and incurring latency and cost issues. Existing approaches tackle such challenges by selectively extracting relevant user data (i.e. selective retrieval) to construct a prompt for downstream tasks. However, retrieval-based methods are limited by potential information loss, lack of more profound user understanding, and cold-start challenges. To overcome these limitations, we propose a novel summary-augmented approach by extending retrieval-augmented personalization with task-aware user summaries generated by LLMs. The summaries can be generated and stored offline, enabling real-world systems with runtime constraints like voice assistants to leverage the power of LLMs. Experiments show our method with 75% less of retrieved user data is on-par or outperforms retrieval augmentation on most tasks in the LaMP personalization benchmark. We demonstrate that offline summarization via LLMs and runtime retrieval enables better performance for personalization on a range of tasks under practical constraints.
△ Less
Submitted 30 October, 2023;
originally announced October 2023.
-
SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback
Authors:
Christopher Richardson,
Anirudh Sundar,
Larry Heck
Abstract:
Commonsense reasoning is a critical aspect of human communication. Despite recent advances in conversational AI driven by large language models, commonsense reasoning remains a challenging task. In this work, we introduce SYNDICOM - a method for improving commonsense in dialogue response generation. SYNDICOM consists of two components. The first component is a dataset composed of commonsense dialo…
▽ More
Commonsense reasoning is a critical aspect of human communication. Despite recent advances in conversational AI driven by large language models, commonsense reasoning remains a challenging task. In this work, we introduce SYNDICOM - a method for improving commonsense in dialogue response generation. SYNDICOM consists of two components. The first component is a dataset composed of commonsense dialogues created from a knowledge graph and synthesized into natural language. This dataset includes both valid and invalid responses to dialogue contexts, along with natural language feedback (NLF) for the invalid responses. The second contribution is a two-step procedure: training a model to predict natural language feedback (NLF) for invalid responses, and then training a response generation model conditioned on the predicted NLF, the invalid response, and the dialogue. SYNDICOM is scalable and does not require reinforcement learning. Empirical results on three tasks are evaluated using a broad range of metrics. SYNDICOM achieves a relative improvement of 53% over ChatGPT on ROUGE1, and human evaluators prefer SYNDICOM over ChatGPT 57% of the time. We will publicly release the code and the full dataset.
△ Less
Submitted 18 September, 2023;
originally announced September 2023.
-
Predicting Adverse Neonatal Outcomes for Preterm Neonates with Multi-Task Learning
Authors:
Jingyang Lin,
Junyu Chen,
Hanjia Lyu,
Igor Khodak,
Divya Chhabra,
Colby L Day Richardson,
Irina Prelipcean,
Andrew M Dylag,
Jiebo Luo
Abstract:
Diagnosis of adverse neonatal outcomes is crucial for preterm survival since it enables doctors to provide timely treatment. Machine learning (ML) algorithms have been demonstrated to be effective in predicting adverse neonatal outcomes. However, most previous ML-based methods have only focused on predicting a single outcome, ignoring the potential correlations between different outcomes, and pote…
▽ More
Diagnosis of adverse neonatal outcomes is crucial for preterm survival since it enables doctors to provide timely treatment. Machine learning (ML) algorithms have been demonstrated to be effective in predicting adverse neonatal outcomes. However, most previous ML-based methods have only focused on predicting a single outcome, ignoring the potential correlations between different outcomes, and potentially leading to suboptimal results and overfitting issues. In this work, we first analyze the correlations between three adverse neonatal outcomes and then formulate the diagnosis of multiple neonatal outcomes as a multi-task learning (MTL) problem. We then propose an MTL framework to jointly predict multiple adverse neonatal outcomes. In particular, the MTL framework contains shared hidden layers and multiple task-specific branches. Extensive experiments have been conducted using Electronic Health Records (EHRs) from 121 preterm neonates. Empirical results demonstrate the effectiveness of the MTL framework. Furthermore, the feature importance is analyzed for each neonatal outcome, providing insights into model interpretability.
△ Less
Submitted 27 March, 2023;
originally announced March 2023.
-
Learning to Retrieve Engaging Follow-Up Queries
Authors:
Christopher Richardson,
Sudipta Kar,
Anjishnu Kumar,
Anand Ramachandran,
Omar Zia Khan,
Zeynab Raeesy,
Abhinav Sethy
Abstract:
Open domain conversational agents can answer a broad range of targeted queries. However, the sequential nature of interaction with these systems makes knowledge exploration a lengthy task which burdens the user with asking a chain of well phrased questions. In this paper, we present a retrieval based system and associated dataset for predicting the next questions that the user might have. Such a s…
▽ More
Open domain conversational agents can answer a broad range of targeted queries. However, the sequential nature of interaction with these systems makes knowledge exploration a lengthy task which burdens the user with asking a chain of well phrased questions. In this paper, we present a retrieval based system and associated dataset for predicting the next questions that the user might have. Such a system can proactively assist users in knowledge exploration leading to a more engaging dialog. The retrieval system is trained on a dataset which contains ~14K multi-turn information-seeking conversations with a valid follow-up question and a set of invalid candidates. The invalid candidates are generated to simulate various syntactic and semantic confounders such as paraphrases, partial entity match, irrelevant entity, and ASR errors. We use confounder specific techniques to simulate these negative examples on the OR-QuAC dataset and develop a dataset called the Follow-up Query Bank (FQ-Bank). Then, we train ranking models on FQ-Bank and present results comparing supervised and unsupervised approaches. The results suggest that we can retrieve the valid follow-ups by ranking them in higher positions compared to confounders, but further knowledge grounding can improve ranking performance.
△ Less
Submitted 21 February, 2023;
originally announced February 2023.
-
Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
Authors:
Christopher Richardson,
Larry Heck
Abstract:
Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has enabled significant advances in conversational AI, including the development of open-dialogue systems capable of coherent, salient conversations which can answer questions, chat casually, and complete tasks. However, state-…
▽ More
Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has enabled significant advances in conversational AI, including the development of open-dialogue systems capable of coherent, salient conversations which can answer questions, chat casually, and complete tasks. However, state-of-the-art models still struggle with tasks that involve higher levels of reasoning - including commonsense reasoning that humans find trivial. This paper presents a survey of recent conversational AI research focused on commonsense reasoning. The paper lists relevant training datasets and describes the primary approaches to include commonsense in conversational AI. The paper also discusses benchmarks used for evaluating commonsense in conversational AI problems. Finally, the paper presents preliminary observations of the limited commonsense capabilities of two state-of-the-art open dialogue models, BlenderBot3 and LaMDA, and its negative effect on natural interactions. These observations further motivate research on commonsense reasoning in conversational AI.
△ Less
Submitted 15 February, 2023;
originally announced February 2023.
-
Not always about you: Prioritizing community needs when developing endangered language technology
Authors:
Zoey Liu,
Crystal Richardson,
Richard Hatcher Jr,
Emily Prud'hommeaux
Abstract:
Languages are classified as low-resource when they lack the quantity of data necessary for training statistical and machine learning tools and models. Causes of resource scarcity vary but can include poor access to technology for developing these resources, a relatively small population of speakers, or a lack of urgency for collecting such resources in bilingual populations where the second langua…
▽ More
Languages are classified as low-resource when they lack the quantity of data necessary for training statistical and machine learning tools and models. Causes of resource scarcity vary but can include poor access to technology for developing these resources, a relatively small population of speakers, or a lack of urgency for collecting such resources in bilingual populations where the second language is high-resource. As a result, the languages described as low-resource in the literature are as different as Finnish on the one hand, with millions of speakers using it in every imaginable domain, and Seneca, with only a small-handful of fluent speakers using the language primarily in a restricted domain. While issues stemming from the lack of resources necessary to train models unite this disparate group of languages, many other issues cut across the divide between widely-spoken low resource languages and endangered languages. In this position paper, we discuss the unique technological, cultural, practical, and ethical challenges that researchers and indigenous speech community members face when working together to develop language technology to support endangered language documentation and revitalization. We report the perspectives of language teachers, Master Speakers and elders from indigenous communities, as well as the point of view of academics. We describe an ongoing fruitful collaboration and make recommendations for future partnerships between academic researchers and language community stakeholders.
△ Less
Submitted 12 April, 2022;
originally announced April 2022.
-
Achieving a quantum smart workforce
Authors:
Clarice D. Aiello,
D. D. Awschalom,
Hannes Bernien,
Tina Brower-Thomas,
Kenneth R. Brown,
Todd A. Brun,
Justin R. Caram,
Eric Chitambar,
Rosa Di Felice,
Michael F. J. Fox,
Stephan Haas,
Alexander W. Holleitner,
Eric R. Hudson,
Jeffrey H. Hunt,
Robert Joynt,
Scott Koziol,
H. J. Lewandowski,
Douglas T. McClure,
Jens Palsberg,
Gina Passante,
Kristen L. Pudenz,
Christopher J. K. Richardson,
Jessica L. Rosenberg,
R. S. Ross,
Mark Saffman
, et al. (7 additional authors not shown)
Abstract:
Interest in building dedicated Quantum Information Science and Engineering (QISE) education programs has greatly expanded in recent years. These programs are inherently convergent, complex, often resource intensive and likely require collaboration with a broad variety of stakeholders. In order to address this combination of challenges, we have captured ideas from many members in the community. Thi…
▽ More
Interest in building dedicated Quantum Information Science and Engineering (QISE) education programs has greatly expanded in recent years. These programs are inherently convergent, complex, often resource intensive and likely require collaboration with a broad variety of stakeholders. In order to address this combination of challenges, we have captured ideas from many members in the community. This manuscript not only addresses policy makers and funding agencies (both public and private and from the regional to the international level) but also contains needs identified by industry leaders and discusses the difficulties inherent in creating an inclusive QISE curriculum. We report on the status of eighteen post-secondary education programs in QISE and provide guidance for building new programs. Lastly, we encourage the development of a comprehensive strategic plan for quantum education and workforce development as a means to make the most of the ongoing substantial investments being made in QISE.
△ Less
Submitted 23 October, 2020;
originally announced October 2020.
-
LEoPart: a particle library for FEniCS
Authors:
Jakob M. Maljaars,
Chris N. Richardson,
Nathan Sime
Abstract:
This paper introduces LEoPart, an add-on for the open source finite element software library FEniCS to seamlessly integrate Lagrangian particle functionality with (Eulerian) mesh-based finite element (FE) approaches. LEoPart - which is so much as to say: `Lagrangian-Eulerian on Particles' - contains tools for efficient, accurate and scalable advection of Lagrangian particles on arbitrary polyhedra…
▽ More
This paper introduces LEoPart, an add-on for the open source finite element software library FEniCS to seamlessly integrate Lagrangian particle functionality with (Eulerian) mesh-based finite element (FE) approaches. LEoPart - which is so much as to say: `Lagrangian-Eulerian on Particles' - contains tools for efficient, accurate and scalable advection of Lagrangian particles on arbitrary polyhedral meshes. In addition, LEoPart comes with several projection operators for exchanging information between the scattered particles and the mesh and \textit{vice versa}. These projection operators are based on a variational framework, which allows extension to high-order accuracy. In particular, by implementing a dedicated PDE-constrained particle-mesh projection operator, LEoPart provides all the tools for diffusion-free advection, while simultaneously achieving optimal convergence and ensuring conservation of the projected particle quantities on the underlying mesh. A range of numerical examples that are prototypical to passive and active tracer methods highlight the properties and the parallel performance of the different tools in LEoPart. Finally, future developments are identified. The source code for LEoPart is actively maintained and available under an open source license at https://bitbucket.org/jakob_maljaars/leopart.
△ Less
Submitted 29 May, 2020; v1 submitted 23 December, 2019;
originally announced December 2019.
-
Abstractions and automated algorithms for mixed domain finite element methods
Authors:
Cécile Daversin-Catty,
Chris N. Richardson,
Ada J. Ellingsrud,
Marie E. Rognes
Abstract:
Mixed dimensional partial differential equations (PDEs) are equations coupling unknown fields defined over domains of differing topological dimension. Such equations naturally arise in a wide range of scientific fields including geology, physiology, biology and fracture mechanics. Mixed dimensional PDEs are also commonly encountered when imposing non-standard conditions over a subspace of lower di…
▽ More
Mixed dimensional partial differential equations (PDEs) are equations coupling unknown fields defined over domains of differing topological dimension. Such equations naturally arise in a wide range of scientific fields including geology, physiology, biology and fracture mechanics. Mixed dimensional PDEs are also commonly encountered when imposing non-standard conditions over a subspace of lower dimension e.g. through a Lagrange multiplier. In this paper, we present general abstractions and algorithms for finite element discretizations of mixed domain and mixed dimensional PDEs of co-dimension up to one (i.e. nD-mD with |n-m| <= 1). We introduce high level mathematical software abstractions together with lower level algorithms for expressing and efficiently solving such coupled systems. The concepts introduced here have also been implemented in the context of the FEniCS finite element software. We illustrate the new features through a range of examples, including a constrained Poisson problem, a set of Stokes-type flow models and a model for ionic electrodiffusion.
△ Less
Submitted 4 November, 2019;
originally announced November 2019.
-
Scalable computation of thermomechanical turbomachinery problems
Authors:
Chris N. Richardson,
Nathan Sime,
Garth N. Wells
Abstract:
A commonly held view in the turbomachinery community is that finite element methods are not well-suited for very large-scale thermomechanical simulations. We seek to dispel this notion by presenting performance data for a collection of realistic, large-scale thermomechanical simulations. We describe the necessary technology to compute problems with $O(10^7)$ to $O(10^9)$ degrees-of-freedom, and em…
▽ More
A commonly held view in the turbomachinery community is that finite element methods are not well-suited for very large-scale thermomechanical simulations. We seek to dispel this notion by presenting performance data for a collection of realistic, large-scale thermomechanical simulations. We describe the necessary technology to compute problems with $O(10^7)$ to $O(10^9)$ degrees-of-freedom, and emphasise what is required to achieve near linear computational complexity with good parallel scaling. Performance data is presented for turbomachinery components with up to 3.3 billion degrees-of-freedom. The software libraries used to perform the simulations are freely available under open source licenses. The performance demonstrated in this work opens up the possibility of system-level thermomechanical modelling, and lays the foundation for further research into high-performance formulations for even larger problems and for other physical processes, such as contact, that are important in turbomachinery analysis.
△ Less
Submitted 15 November, 2018; v1 submitted 26 April, 2018;
originally announced April 2018.
-
A high-bias, low-variance introduction to Machine Learning for physicists
Authors:
Pankaj Mehta,
Marin Bukov,
Ching-Hao Wang,
Alexandre G. R. Day,
Clint Richardson,
Charles K. Fisher,
David J. Schwab
Abstract:
Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, r…
▽ More
Machine Learning (ML) is one of the most exciting and dynamic areas of modern research and application. The purpose of this review is to provide an introduction to the core concepts and tools of machine learning in a manner easily understood and intuitive to physicists. The review begins by covering fundamental concepts in ML and modern statistics such as the bias-variance tradeoff, overfitting, regularization, generalization, and gradient descent before moving on to more advanced topics in both supervised and unsupervised learning. Topics covered in the review include ensemble models, deep learning and neural networks, clustering and data visualization, energy-based models (including MaxEnt models and Restricted Boltzmann Machines), and variational methods. Throughout, we emphasize the many natural connections between ML and statistical physics. A notable aspect of the review is the use of Python Jupyter notebooks to introduce modern ML/statistical packages to readers using physics-inspired datasets (the Ising Model and Monte-Carlo simulations of supersymmetric decays of proton-proton collisions). We conclude with an extended outlook discussing possible uses of machine learning for furthering our understanding of the physical world as well as open problems in ML where physicists may be able to contribute. (Notebooks are available at https://physics.bu.edu/~pankajm/MLnotebooks.html )
△ Less
Submitted 27 May, 2019; v1 submitted 23 March, 2018;
originally announced March 2018.
-
Containers for portable, productive and performant scientific computing
Authors:
Jack S. Hale,
Lizao Li,
Chris N. Richardson,
Garth N. Wells
Abstract:
Containers are an emerging technology that hold promise for improving productivity and code portability in scientific computing. We examine Linux container technology for the distribution of a non-trivial scientific computing software stack and its execution on a spectrum of platforms from laptop computers through to high performance computing (HPC) systems. We show on a workstation and a leadersh…
▽ More
Containers are an emerging technology that hold promise for improving productivity and code portability in scientific computing. We examine Linux container technology for the distribution of a non-trivial scientific computing software stack and its execution on a spectrum of platforms from laptop computers through to high performance computing (HPC) systems. We show on a workstation and a leadership-class HPC system that when deployed appropriately there are no performance penalties running scientific programs inside containers. For Python code run on large parallel computers, the run time is reduced inside a container due to faster library imports. The software distribution approach and data that we present will help developers and users decide on whether container technology is appropriate for them. We also provide guidance for the vendors of HPC systems that rely on proprietary libraries for performance on what they can do to make containers work seamlessly and without performance penalty.
△ Less
Submitted 3 November, 2016; v1 submitted 26 August, 2016;
originally announced August 2016.
