-
ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP
Authors:
Philipp Kohl,
Nils Freyer,
Yoka Krämer,
Henri Werth,
Steffen Wolf,
Bodo Kraft,
Matthias Meinecke,
Albert Zündorf
Abstract:
Supervised machine learning and deep learning require a large amount of labeled data, which data scientists obtain in a manual, and time-consuming annotation process. To mitigate this challenge, Active Learning (AL) proposes promising data points to annotators they annotate next instead of a subsequent or random sample. This method is supposed to save annotation effort while maintaining model perf…
▽ More
Supervised machine learning and deep learning require a large amount of labeled data, which data scientists obtain in a manual, and time-consuming annotation process. To mitigate this challenge, Active Learning (AL) proposes promising data points to annotators they annotate next instead of a subsequent or random sample. This method is supposed to save annotation effort while maintaining model performance. However, practitioners face many AL strategies for different tasks and need an empirical basis to choose between them. Surveys categorize AL strategies into taxonomies without performance indications. Presentations of novel AL strategies compare the performance to a small subset of strategies. Our contribution addresses the empirical basis by introducing a reproducible active learning evaluation (ALE) framework for the comparative evaluation of AL strategies in NLP. The framework allows the implementation of AL strategies with low effort and a fair data-driven comparison through defining and tracking experiment parameters (e.g., initial dataset size, number of data points per query step, and the budget). ALE helps practitioners to make more informed decisions, and researchers can focus on developing new, effective AL strategies and deriving best practices for specific use cases. With best practices, practitioners can lower their annotation costs. We present a case study to illustrate how to use the framework.
△ Less
Submitted 1 August, 2023;
originally announced August 2023.
-
Explaining Relation Classification Models with Semantic Extents
Authors:
Lars Klöser,
Andre Büsgen,
Philipp Kohl,
Bodo Kraft,
Albert Zündorf
Abstract:
In recent years, the development of large pretrained language models, such as BERT and GPT, significantly improved information extraction systems on various tasks, including relation classification. State-of-the-art systems are highly accurate on scientific benchmarks. A lack of explainability is currently a complicating factor in many real-world applications. Comprehensible systems are necessary…
▽ More
In recent years, the development of large pretrained language models, such as BERT and GPT, significantly improved information extraction systems on various tasks, including relation classification. State-of-the-art systems are highly accurate on scientific benchmarks. A lack of explainability is currently a complicating factor in many real-world applications. Comprehensible systems are necessary to prevent biased, counterintuitive, or harmful decisions.
We introduce semantic extents, a concept to analyze decision patterns for the relation classification task. Semantic extents are the most influential parts of texts concerning classification decisions. Our definition allows similar procedures to determine semantic extents for humans and models. We provide an annotation tool and a software framework to determine semantic extents for humans and models conveniently and reproducibly. Comparing both reveals that models tend to learn shortcut patterns from data. These patterns are hard to detect with current interpretability methods, such as input reductions. Our approach can help detect and eliminate spurious decision patterns during model development. Semantic extents can increase the reliability and security of natural language processing systems. Semantic extents are an essential step in enabling applications in critical areas like healthcare or finance. Moreover, our work opens new research directions for developing methods to explain deep learning models.
△ Less
Submitted 4 August, 2023;
originally announced August 2023.
-
Multi-Attribute Relation Extraction (MARE) -- Simplifying the Application of Relation Extraction
Authors:
Lars Klöser,
Philipp Kohl,
Bodo Kraft,
Albert Zündorf
Abstract:
Natural language understanding's relation extraction makes innovative and encouraging novel business concepts possible and facilitates new digitilized decision-making processes. Current approaches allow the extraction of relations with a fixed number of entities as attributes. Extracting relations with an arbitrary amount of attributes requires complex systems and costly relation-trigger annotatio…
▽ More
Natural language understanding's relation extraction makes innovative and encouraging novel business concepts possible and facilitates new digitilized decision-making processes. Current approaches allow the extraction of relations with a fixed number of entities as attributes. Extracting relations with an arbitrary amount of attributes requires complex systems and costly relation-trigger annotations to assist these systems. We introduce multi-attribute relation extraction (MARE) as an assumption-less problem formulation with two approaches, facilitating an explicit mapping from business use cases to the data annotations. Avoiding elaborated annotation constraints simplifies the application of relation extraction approaches. The evaluation compares our models to current state-of-the-art event extraction and binary relation extraction methods. Our approaches show improvement compared to these on the extraction of general multi-attribute relations.
△ Less
Submitted 17 November, 2021;
originally announced November 2021.
-
STAMP 4 NLP -- An Agile Framework for Rapid Quality-Driven NLP Applications Development
Authors:
Philipp Kohl,
Oliver Schmidts,
Lars Klöser,
Henri Werth,
Bodo Kraft,
Albert Zündorf
Abstract:
The progress in natural language processing (NLP) research over the last years, offers novel business opportunities for companies, as automated user interaction or improved data analysis. Building sophisticated NLP applications requires dealing with modern machine learning (ML) technologies, which impedes enterprises from establishing successful NLP projects. Our experience in applied NLP research…
▽ More
The progress in natural language processing (NLP) research over the last years, offers novel business opportunities for companies, as automated user interaction or improved data analysis. Building sophisticated NLP applications requires dealing with modern machine learning (ML) technologies, which impedes enterprises from establishing successful NLP projects. Our experience in applied NLP research projects shows that the continuous integration of research prototypes in production-like environments with quality assurance builds trust in the software and shows convenience and usefulness regarding the business goal. We introduce STAMP 4 NLP as an iterative and incremental process model for developing NLP applications. With STAMP 4 NLP, we merge software engineering principles with best practices from data science. Instantiating our process model allows efficiently creating prototypes by utilizing templates, conventions, and implementations, enabling developers and data scientists to focus on the business goals. Due to our iterative-incremental approach, businesses can deploy an enhanced version of the prototype to their software environment after every iteration, maximizing potential business value and trust early and avoiding the cost of successful yet never deployed experiments.
△ Less
Submitted 16 November, 2021;
originally announced November 2021.
-
Applying Model-Driven Engineering to Stimulate the Adoption of DevOps Processes in Small and Medium-Sized Development Organizations
Authors:
Jonas Sorgalla,
Philip Wizenty,
Florian Rademacher,
Sabine Sachweh,
Albert Zündorf
Abstract:
Purpose: Microservice Architecture (MSA) denotes an increasingly popular architectural style in which business capabilities are wrapped into autonomously developable and deployable software components called microservices. Microservice applications are developed by multiple DevOps teams each owning one or more services. In this article, we explore the state of how DevOps teams in small and medium-…
▽ More
Purpose: Microservice Architecture (MSA) denotes an increasingly popular architectural style in which business capabilities are wrapped into autonomously developable and deployable software components called microservices. Microservice applications are developed by multiple DevOps teams each owning one or more services. In this article, we explore the state of how DevOps teams in small and medium-sized organizations (SMOs) cope with MSA and how they can be supported.
Methods: We show through a secondary analysis of an exploratory interview study comprising six cases, that the organizational and technological complexity resulting from MSA poses particular challenges for small and medium-sized organizations (SMOs). We apply Model-Driven Engineering to address these challenges.
Results: As results of the second analysis, we identify the challenge areas of building and maintaining a common architectural understanding, and dealing with deployment technologies. To support DevOps teams of SMOs in coping with these challenges, we present a model-driven workflow based on LEMMA - the Language Ecosystem for Modeling Microservice Architecture. To implement the workflow, we extend LEMMA with the functionality to (i) generate models from API documentation; (ii) reference remote models owned by other teams; (iii) generate deployment specifications; and (iv) generate a visual representation of the overall architecture.
Conclusion: We validate the model-driven workflow and our extensions to LEMMA through a case study showing that the added functionality to LEMMA can bring efficiency gains for DevOps teams. To develop best practices for applying our workflow to maximize efficiency in SMOs, we plan to conduct more empirical research in the field in the future.
△ Less
Submitted 26 July, 2021;
originally announced July 2021.
-
Commutative Event Sourcing vs. Triple Graph Grammars
Authors:
Sebastian Copei,
Albert Zündorf
Abstract:
This paper proposes Commutative Event Sourcing as a simple and reliable mechanism for model synchronisation, bidirectional model to model transformations, incremental updates, and collaborative editing. Commutative Event Sourcing is a restricted form of a Triple Graph Grammar where the rules or editing commands are either overwriting or commutative. This restriction gets rid of a lot of Triple Gra…
▽ More
This paper proposes Commutative Event Sourcing as a simple and reliable mechanism for model synchronisation, bidirectional model to model transformations, incremental updates, and collaborative editing. Commutative Event Sourcing is a restricted form of a Triple Graph Grammar where the rules or editing commands are either overwriting or commutative. This restriction gets rid of a lot of Triple Graph Grammar complexity and it becomes possible to implement model synchronisation manually. Thus, you are not restricted to Java as your programming language and you do not need to use a proprietary library, framework, or tool. You do not even have to dig into graph grammar theory.
△ Less
Submitted 21 January, 2021;
originally announced January 2021.
-
The Fulib Solution to the TTC 2020 Migration Case
Authors:
Sebastian Copei,
Albert Zündorf
Abstract:
At Kassel University we are working on a solution for bidirectional transformations based on event sourcing for about a year, now. It turned out, that the TTC 2020 migration case is a special case of a bidirectional transformation and that our approach provides a reasonable solution for it.
At Kassel University we are working on a solution for bidirectional transformations based on event sourcing for about a year, now. It turned out, that the TTC 2020 migration case is a special case of a bidirectional transformation and that our approach provides a reasonable solution for it.
△ Less
Submitted 9 December, 2020;
originally announced December 2020.
-
On Collaborative Model-driven Development of Microservices
Authors:
Jonas Sorgalla,
Florian Rademacher,
Sabine Sachweh,
Albert Zündorf
Abstract:
Microservice Architecture (MSA) denotes an emerging architectural style for distributed and service-based systems whereby each microservice is highly cohesive and implements a single business capability. A microservice system consists of multiple, loosely coupled microservices. It provides complex capabilities through services interacting in choreographies. A single dedicated team, typically pract…
▽ More
Microservice Architecture (MSA) denotes an emerging architectural style for distributed and service-based systems whereby each microservice is highly cohesive and implements a single business capability. A microservice system consists of multiple, loosely coupled microservices. It provides complex capabilities through services interacting in choreographies. A single dedicated team, typically practicing DevOps, is responsible for each microservice, i.e., it "owns" the service. However, while systems relying on MSA have several architectural advantages especially for cloud applications, their realization is characterized by an increased accidental complexity due to redundant handcrafting of implementation, e.g., to make each service standalone runnable. A promising way to cope with such complexity is the usage of Model-driven Development (MDD) whereby models are used as first-class entities in the software development process. Although there are already first steps taken on how MDD could be applied by a single team to implement its microservices, the question of how MDD can be adapted to MSA's development distribution across multiple teams remains an issue. In this paper we envision the application of Collaborative Model-driven Software Engineering (CMDSE) to MDD of MSA by surveying relevant characteristics of CMDSE and identifying challenges for its application to MSA. The present paper takes a first step towards enabling holistic MDD of MSA across microservice teams.
△ Less
Submitted 3 May, 2018;
originally announced May 2018.
-
Towards a Viewpoint-specific Metamodel for Model-driven Development of Microservice Architecture
Authors:
Florian Rademacher,
Jonas Sorgalla,
Sabine Sachweh,
Albert Zündorf
Abstract:
Microservice Architecture (MSA) is a service-based architectural style with a strong emphasis on high cohesion and loose coupling. It is commonly regarded as a descendant of Service-oriented Architecture (SOA) and thus might draw on existing findings of SOA research. This paper presents a metamodel for Model-driven Development (MDD) of MSA, which is deduced from existing SOA modeling approaches, b…
▽ More
Microservice Architecture (MSA) is a service-based architectural style with a strong emphasis on high cohesion and loose coupling. It is commonly regarded as a descendant of Service-oriented Architecture (SOA) and thus might draw on existing findings of SOA research. This paper presents a metamodel for Model-driven Development (MDD) of MSA, which is deduced from existing SOA modeling approaches, but also incorporates MSA-specific modeling concepts. It is divided into the three viewpoints Data, Service and Operation, each of which encapsulates concepts related to a certain aspect of MSA. The metamodel aims to support DevOps-based MSA development and automatic transformation of metamodel instances into MSA implementations.
△ Less
Submitted 26 April, 2018;
originally announced April 2018.
-
Analysis of Service-oriented Modeling Approaches for Viewpoint-specific Model-driven Development of Microservice Architecture
Authors:
Florian Rademacher,
Sabine Sachweh,
Albert Zündorf
Abstract:
Microservice Architecture (MSA) is a novel service-based architectural style for distributed software systems. Compared to Service-oriented Architecture (SOA), MSA puts a stronger focus on self-containment of services. Each microservice is responsible for realizing exactly one business or technological capability that is distinct from other services' capabilities. Additionally, on the implementati…
▽ More
Microservice Architecture (MSA) is a novel service-based architectural style for distributed software systems. Compared to Service-oriented Architecture (SOA), MSA puts a stronger focus on self-containment of services. Each microservice is responsible for realizing exactly one business or technological capability that is distinct from other services' capabilities. Additionally, on the implementation and operation level, microservices are self-contained in that they are developed, tested, deployed and operated independently from each other. Next to these characteristics that distinguish MSA from SOA, both architectural styles rely on services as building blocks of distributed software architecture and hence face similar challenges regarding, e.g., service identification, composition and provisioning. However, in contrast to MSA, SOA may rely on an extensive body of knowledge to tackle these challenges. Thus, due to both architectural styles being service-based, the question arises to what degree MSA might draw on existing findings of SOA research and practice. In this paper we address this question in the field of Model-driven Development (MDD) for design and operation of service-based architectures. Therefore, we present an analysis of existing MDD approaches to SOA, which comprises the identification and semantic clustering of modeling concepts for SOA design and operation. For each concept cluster, the analysis assesses its applicability to MDD of MSA (MSA-MDD) and assigns it to a specific modeling viewpoint. The goal of the presented analysis is to provide a conceptual foundation for an MSA-MDD metamodel.
△ Less
Submitted 26 April, 2018;
originally announced April 2018.
