-
Is Our Chatbot Telling Lies? Assessing Correctness of an LLM-based Dutch Support Chatbot
Authors:
Herman Lassche,
Michiel Overeem,
Ayushi Rastogi
Abstract:
Companies support their customers using live chats and chatbots to gain their loyalty. AFAS is a Dutch company aiming to leverage the opportunity large language models (LLMs) offer to answer customer queries with minimal to no input from its customer support team. Adding to its complexity, it is unclear what makes a response correct, and that too in Dutch. Further, with minimal data available for…
▽ More
Companies support their customers using live chats and chatbots to gain their loyalty. AFAS is a Dutch company aiming to leverage the opportunity large language models (LLMs) offer to answer customer queries with minimal to no input from its customer support team. Adding to its complexity, it is unclear what makes a response correct, and that too in Dutch. Further, with minimal data available for training, the challenge is to identify whether an answer generated by a large language model is correct and do it on the fly.
This study is the first to define the correctness of a response based on how the support team at AFAS makes decisions. It leverages literature on natural language generation and automated answer grading systems to automate the decision-making of the customer support team. We investigated questions requiring a binary response (e.g., Would it be possible to adjust tax rates manually?) or instructions (e.g., How would I adjust tax rate manually?) to test how close our automated approach reaches support rating. Our approach can identify wrong messages in 55\% of the cases. This work shows the viability of automatically assessing when our chatbot tell lies.
△ Less
Submitted 29 October, 2024;
originally announced November 2024.
-
An Empirical Characterization of Event Sourced Systems and Their Schema Evolution -- Lessons from Industry
Authors:
Michiel Overeem,
Marten Spoor,
Slinger Jansen,
Sjaak Brinkkemper
Abstract:
Event sourced systems are increasing in popularity because they are reliable, flexible, and scalable. In this article, we point a microscope at a software architecture pattern that is rapidly gaining popularity in industry, but has not received as much attention from the scientific community. We do so through constructivist grounded theory, which proves a suitable qualitative method for extracting…
▽ More
Event sourced systems are increasing in popularity because they are reliable, flexible, and scalable. In this article, we point a microscope at a software architecture pattern that is rapidly gaining popularity in industry, but has not received as much attention from the scientific community. We do so through constructivist grounded theory, which proves a suitable qualitative method for extracting architectural knowledge from practitioners. Based on the discussion of 19 event sourced systems we explore the rationale for and the context of the event sourcing pattern. A description of the pattern itself and its relation to other patterns as discussed with practitioners is given. The description itself is grounded in the experience of 25 engineers, making it a reliable source for both new practitioners and scientists. We identify five challenges that practitioners experience: event system evolution, the steep learning curve, lack of available technology, rebuilding projections, and data privacy. For the first challenge of event system evolution, we uncover five tactics and solutions that support practitioners in their design choices when developing evolving event sourced systems: versioned events, weak schema, upcasting, in-place transformation, and copy-and-transform.
△ Less
Submitted 2 April, 2021;
originally announced April 2021.
-
Source Data for the Focus Area Maturity Model for API Management
Authors:
Max Mathijssen,
Michiel Overeem,
Slinger Jansen
Abstract:
We define API Management as an activity that enables organizations to design, publish and deploy their APIs for (external) developers to consume. API Management capabilities such as controlling API lifecycles, access and authentication to APIs, monitoring, throttling and analyzing API usage, as well as providing security and documentation. These capabilities are often implemented through an integr…
▽ More
We define API Management as an activity that enables organizations to design, publish and deploy their APIs for (external) developers to consume. API Management capabilities such as controlling API lifecycles, access and authentication to APIs, monitoring, throttling and analyzing API usage, as well as providing security and documentation. These capabilities are often implemented through an integrated platform. This data set describes the API Management Focus Area Maturity Model (API-m-FAMM). In a structured manner, this model aims to support organizations that expose their API(s) to third-party developers in their API management activities. Through a thorough Systematic Literature Review (SLR), 114 practices and 39 capabilities were collected. Subsequently, these practices and capabilities were categorized into 6 focus areas. Next, the practices and capabilities were analyzed and verified through inter-rater agreement and four validation sessions with all involved researchers. Then, the collection of practices and capabilities was verified by using information gathered from supplemental literature, online blog posts, websites, commercial API management platform documentation and third-party tooling. As a result, the initial body of practices and capabilities was narrowed down to 87 practices and 23 capabilities. These practices are described by a practice code, name, description, conditions for implementation, the role responsible for the practice, and the associated literature in which the practice was originally identified. Capabilities and focus areas are described by a code, description and, optionally, the associated literature in which it was originally identified. Using the API-m-FAMM, organizations may evaluate, improve upon and assess the degree of maturity their business processes regarding the topic of API management have.
△ Less
Submitted 27 May, 2021; v1 submitted 21 July, 2020;
originally announced July 2020.
-
Identification of Practices and Capabilities in API Management: A Systematic Literature Review
Authors:
Max Mathijssen,
Michiel Overeem,
Slinger Jansen
Abstract:
Traditional organizations are increasingly becoming software producing organizations. This software is enabling them to integrate business processes between different departments and with other organizations through Application Programming Interfaces (APIs). The main task of managing APIs is to ensure that the APIs are easy to use by third parties, such as providing helpful documentation, monitori…
▽ More
Traditional organizations are increasingly becoming software producing organizations. This software is enabling them to integrate business processes between different departments and with other organizations through Application Programming Interfaces (APIs). The main task of managing APIs is to ensure that the APIs are easy to use by third parties, such as providing helpful documentation, monitoring API performance, and even monetizing API usage. The knowledge on API management is scattered across academic literature. In this document, we describe a Systematic Literature Review (SLR) that has the goal of collecting API Management practices and capabilities related to API Management, as well as proposing a comprehensive definition of the topic. In the scope of this work, a practice is defined as any practice that has the express goal to improve, encourage and manage the usage of APIs. Capabilities are defined as the ability to achieve a certain goal related to API Management, through the execution of two or more interrelated practices. We follow a standard method for SLRs in software engineering. We managed to collect 24 unique definitions for the topic, 114 practices and 39 capabilities.
△ Less
Submitted 18 June, 2020;
originally announced June 2020.
