-
"Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems
Authors:
Mireia Hernandez Caralt,
Ivan Sekulić,
Filip Carević,
Nghia Khau,
Diana Nicoleta Popa,
Bruna Guedes,
Victor Guimarães,
Zeyu Yang,
Andre Manso,
Meghana Reddy,
Paolo Rosso,
Roland Mathis
Abstract:
Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a…
▽ More
Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16\% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners.
△ Less
Submitted 10 January, 2025; v1 submitted 26 November, 2024;
originally announced November 2024.
-
SurveySum: A Dataset for Summarizing Multiple Scientific Articles into a Survey Section
Authors:
Leandro Carísio Fernandes,
Gustavo Bartz Guedes,
Thiago Soares Laitz,
Thales Sales Almeida,
Rodrigo Nogueira,
Roberto Lotufo,
Jayr Pereira
Abstract:
Document summarization is a task to shorten texts into concise and informative summaries. This paper introduces a novel dataset designed for summarizing multiple scientific articles into a section of a survey. Our contributions are: (1) SurveySum, a new dataset addressing the gap in domain-specific summarization tools; (2) two specific pipelines to summarize scientific articles into a section of a…
▽ More
Document summarization is a task to shorten texts into concise and informative summaries. This paper introduces a novel dataset designed for summarizing multiple scientific articles into a section of a survey. Our contributions are: (1) SurveySum, a new dataset addressing the gap in domain-specific summarization tools; (2) two specific pipelines to summarize scientific articles into a section of a survey; and (3) the evaluation of these pipelines using multiple metrics to compare their performance. Our results highlight the importance of high-quality retrieval stages and the impact of different configurations on the quality of generated summaries.
△ Less
Submitted 29 August, 2024;
originally announced August 2024.
-
Classification and Clustering of Sentence-Level Embeddings of Scientific Articles Generated by Contrastive Learning
Authors:
Gustavo Bartz Guedes,
Ana Estela Antunes da Silva
Abstract:
Scientific articles are long text documents organized into sections, each describing aspects of the research. Analyzing scientific production has become progressively challenging due to the increase in the number of available articles. Within this scenario, our approach consisted of fine-tuning transformer language models to generate sentence-level embeddings from scientific articles, considering…
▽ More
Scientific articles are long text documents organized into sections, each describing aspects of the research. Analyzing scientific production has become progressively challenging due to the increase in the number of available articles. Within this scenario, our approach consisted of fine-tuning transformer language models to generate sentence-level embeddings from scientific articles, considering the following labels: background, objective, methods, results, and conclusion. We trained our models on three datasets with contrastive learning. Two datasets are from the article's abstracts in the computer science and medical domains. Also, we introduce PMC-Sents-FULL, a novel dataset of sentences extracted from the full texts of medical articles. We compare the fine-tuned and baseline models in clustering and classification tasks to evaluate our approach. On average, clustering agreement measures values were five times higher. For the classification measures, in the best-case scenario, we had an average improvement in F1-micro of 30.73\%. Results show that fine-tuning sentence transformers with contrastive learning and using the generated embeddings in downstream tasks is a feasible approach to sentence classification in scientific articles. Our experiment codes are available on GitHub.
△ Less
Submitted 29 March, 2024;
originally announced April 2024.
-
Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
Authors:
Ivan Sekulić,
Silvia Terragni,
Victor Guimarães,
Nghia Khau,
Bruna Guedes,
Modestas Filipavicius,
André Ferreira Manso,
Roland Mathis
Abstract:
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based me…
▽ More
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.
△ Less
Submitted 20 February, 2024;
originally announced February 2024.
-
In-Context Learning User Simulators for Task-Oriented Dialog Systems
Authors:
Silvia Terragni,
Modestas Filipavicius,
Nghia Khau,
Bruna Guedes,
André Manso,
Roland Mathis
Abstract:
This paper presents a novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples. Unlike traditional simulators, this method eliminates the need for labor-intensive…
▽ More
This paper presents a novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples. Unlike traditional simulators, this method eliminates the need for labor-intensive rule definition or extensive annotated data, making it more efficient and accessible. Additionally, an error analysis of the interaction between the user simulator and dialog system uncovers common mistakes, providing valuable insights into areas that require improvement. Our implementation is available at https://github.com/telepathylabsai/prompt-based-user-simulator.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
Semantic Scene Completion Combining Colour and Depth: preliminary experiments
Authors:
Andre Bernardes Soares Guedes,
Teofilo Emidio de Campos,
Adrian Hilton
Abstract:
Semantic scene completion is the task of producing a complete 3D voxel representation of volumetric occupancy with semantic labels for a scene from a single-view observation. We built upon the recent work of Song et al. (CVPR 2017), who proposed SSCnet, a method that performs scene completion and semantic labelling in a single end-to-end 3D convolutional network. SSCnet uses only depth maps as inp…
▽ More
Semantic scene completion is the task of producing a complete 3D voxel representation of volumetric occupancy with semantic labels for a scene from a single-view observation. We built upon the recent work of Song et al. (CVPR 2017), who proposed SSCnet, a method that performs scene completion and semantic labelling in a single end-to-end 3D convolutional network. SSCnet uses only depth maps as input, even though depth maps are usually obtained from devices that also capture colour information, such as RGBD sensors and stereo cameras. In this work, we investigate the potential of the RGB colour channels to improve SSCnet.
△ Less
Submitted 13 February, 2018;
originally announced February 2018.
-
Quantum Communication Complexity of Quantum Authentication Protocols
Authors:
Elloá B. Guedes,
Francisco Marcos de Assis
Abstract:
In order to perform Quantum Cryptography procedures it is often essencial to ensure that the parties of the communication are authentic. Such task is accomplished by quantum authentication protocols which are distributed algorithms based on the intrinsic properties of Quantum Mechanics. The choice of an authentication protocol must consider that quantum states are very delicate and that the channe…
▽ More
In order to perform Quantum Cryptography procedures it is often essencial to ensure that the parties of the communication are authentic. Such task is accomplished by quantum authentication protocols which are distributed algorithms based on the intrinsic properties of Quantum Mechanics. The choice of an authentication protocol must consider that quantum states are very delicate and that the channel is subject to eavesdropping. However, even in face of the various existing definitions of quantum authentication protocols in the literature, little is known about them in this perspective, and this lack of knowledge may unfavor comparisons and wise choices. In the attempt to overcome this limitation, in the present work we aim at showing an approach to evaluate quantum authentication protocols based on the determination of their quantum communication complexity. Based on our investigation, no similar methods to analyze quantum authentication protocols were found in the literature. Pursuing this further, our approach has advantages that need to be highlighted: it characterizes a systematic procedure to evaluate quantum authentication protocols; its evaluation is intuitive, based only on the protocol execution; the resulting measure is a concise notation of what resources a quantum authentication protocol demands and how many communications are performed; it allows comparisons between protocols; it makes possible to analyze the communication effort when an eavesdropping occurs; and, lastly, it is likely to be applied in almost any quantum authentication protocol. To illustrate the proposed approach, we also bring results about its application in ten existing quantum authentication protocols (data origin authentication and identity authentication). Such evaluations increase the knowledge about the existing protocols, presenting its advantages, limitations and contrasts.
△ Less
Submitted 26 May, 2011;
originally announced May 2011.
-
Examples of the Generalized Quantum Permanent Compromise Attack to the Blum-Micali Construction
Authors:
Elloá B. Guedes,
Francisco Marcos de Assis,
Bernardo Lula Jr
Abstract:
This paper presents examples of the quantum permanent compromise attack to the Blum-Micali construction. Such attacks illustrate how a previous attack to the Blum-Micali generator can be extended to the whole Blum-Micali construction, including the Blum-Blum-Shub and Kaliski generators.
This paper presents examples of the quantum permanent compromise attack to the Blum-Micali construction. Such attacks illustrate how a previous attack to the Blum-Micali generator can be extended to the whole Blum-Micali construction, including the Blum-Blum-Shub and Kaliski generators.
△ Less
Submitted 8 December, 2010;
originally announced December 2010.
