-
How (Not) To Write a Software Engineering Abstract
Authors:
Lutz Prechelt,
Lloyd Montgomery,
Julian Frattini,
Franz Zieris
Abstract:
Background: Abstracts are a particularly valuable element in a software engineering research article. However, not all abstracts are as informative as they could be. Objective: Characterize the structure of abstracts in high-quality software engineering venues. Observe and quantify deficiencies. Suggest guidelines for writing informative abstracts. Methods: Use qualitative open coding to derive co…
▽ More
Background: Abstracts are a particularly valuable element in a software engineering research article. However, not all abstracts are as informative as they could be. Objective: Characterize the structure of abstracts in high-quality software engineering venues. Observe and quantify deficiencies. Suggest guidelines for writing informative abstracts. Methods: Use qualitative open coding to derive concepts that explain relevant properties of abstracts. Identify the archetypical structure of abstracts. Use quantitative content analysis to objectively characterize abstract structure of a sample of 362 abstracts from five presumably high-quality venues. Use exploratory data analysis to find recurring issues in abstracts. Compare the archetypical structure to actual structures. Infer guidelines for producing informative abstracts. Results: Only 29% of the sampled abstracts are complete, i.e., provide background, objective, method, result, and conclusion information. For structured abstracts, the ratio is twice as big. Only 4% of the abstracts are proper, i.e., they also have good readability (Flesch-Kincaid score) and have no informativeness gaps, understandability gaps, nor highly ambiguous sentences. Conclusions: (1) Even in top venues, a large majority of abstracts are far from ideal. (2) Structured abstracts tend to be better than unstructured ones. (3) Artifact-centric works need a different structured format. (4) The community should start requiring conclusions that generalize, which currently are often missing in abstracts.
△ Less
Submitted 25 June, 2025;
originally announced June 2025.
-
Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline
Authors:
Julian Frattini,
Lloyd Montgomery,
Davide Fucci,
Michael Unterkalmsteiner,
Daniel Mendez,
Jannik Fischbach
Abstract:
Requirements quality research, which is dedicated to assessing and improving the quality of requirements specifications, is dependent on research artifacts like data sets (containing information about quality defects) and implementations (automatically detecting and removing these defects). However, recent research exposed that the majority of these research artifacts have become unavailable or ha…
▽ More
Requirements quality research, which is dedicated to assessing and improving the quality of requirements specifications, is dependent on research artifacts like data sets (containing information about quality defects) and implementations (automatically detecting and removing these defects). However, recent research exposed that the majority of these research artifacts have become unavailable or have never been disclosed, which inhibits progress in the research domain. In this work, we aim to improve the availability of research artifacts in requirements quality research. To this end, we (1) extend an artifact recovery initiative, (2) empirically evaluate the reasons for artifact unavailability using Bayesian data analysis, and (3) compile a concise guideline for open science artifact disclosure. Our results include 10 recovered data sets and 7 recovered implementations, empirical support for artifact availability improving over time and the positive effect of public hosting services, and a pragmatic artifact management guideline open for community comments. With this work, we hope to encourage and support adherence to open science principles and improve the availability of research artifacts for the requirements research quality community.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Mining Issue Trackers: Concepts and Techniques
Authors:
Lloyd Montgomery,
Clara Lüders,
Walid Maalej
Abstract:
An issue tracker is a software tool used by organisations to interact with users and manage various aspects of the software development lifecycle. With the rise of agile methodologies, issue trackers have become popular in open and closed-source settings alike. Internal and external stakeholders report, manage, and discuss "issues", which represent different information such as requirements and ma…
▽ More
An issue tracker is a software tool used by organisations to interact with users and manage various aspects of the software development lifecycle. With the rise of agile methodologies, issue trackers have become popular in open and closed-source settings alike. Internal and external stakeholders report, manage, and discuss "issues", which represent different information such as requirements and maintenance tasks. Issue trackers can quickly become complex ecosystems, with dozens of projects, hundreds of users, thousands of issues, and often millions of issue evolutions. Finding and understanding the relevant issues for the task at hand and keeping an overview becomes difficult with time. Moreover, managing issue workflows for diverse projects becomes more difficult as organisations grow, and more stakeholders get involved. To help address these difficulties, software and requirements engineering research have suggested automated techniques based on mining issue tracking data. Given the vast amount of textual data in issue trackers, many of these techniques leverage natural language processing. This chapter discusses four major use cases for algorithmically analysing issue data to assist stakeholders with the complexity and heterogeneity of information in issue trackers. The chapter is accompanied by a follow-along demonstration package with JupyterNotebooks.
△ Less
Submitted 11 July, 2024; v1 submitted 8 March, 2024;
originally announced March 2024.
-
Applying Bayesian Data Analysis for Causal Inference about Requirements Quality: A Controlled Experiment
Authors:
Julian Frattini,
Davide Fucci,
Richard Torkar,
Lloyd Montgomery,
Michael Unterkalmsteiner,
Jannik Fischbach,
Daniel Mendez
Abstract:
It is commonly accepted that the quality of requirements specifications impacts subsequent software engineering activities. However, we still lack empirical evidence to support organizations in deciding whether their requirements are good enough or impede subsequent activities. We aim to contribute empirical evidence to the effect that requirements quality defects have on a software engineering ac…
▽ More
It is commonly accepted that the quality of requirements specifications impacts subsequent software engineering activities. However, we still lack empirical evidence to support organizations in deciding whether their requirements are good enough or impede subsequent activities. We aim to contribute empirical evidence to the effect that requirements quality defects have on a software engineering activity that depends on this requirement. We conduct a controlled experiment in which 25 participants from industry and university generate domain models from four natural language requirements containing different quality defects. We evaluate the resulting models using both frequentist and Bayesian data analysis. Contrary to our expectations, our results show that the use of passive voice only has a minor impact on the resulting domain models. The use of ambiguous pronouns, however, shows a strong effect on various properties of the resulting domain models. Most notably, ambiguous pronouns lead to incorrect associations in domain models. Despite being equally advised against by literature and frequentist methods, the Bayesian data analysis shows that the two investigated quality defects have vastly different impacts on software engineering activities and, hence, deserve different levels of attention. Our employed method can be further utilized by researchers to improve reliable, detailed empirical evidence on requirements quality.
△ Less
Submitted 7 January, 2025; v1 submitted 2 January, 2024;
originally announced January 2024.
-
Requirements Quality Research: a harmonized Theory, Evaluation, and Roadmap
Authors:
Julian Frattini,
Lloyd Montgomery,
Jannik Fischbach,
Daniel Mendez,
Davide Fucci,
Michael Unterkalmsteiner
Abstract:
High-quality requirements minimize the risk of propagating defects to later stages of the software development life cycle. Achieving a sufficient level of quality is a major goal of requirements engineering. This requires a clear definition and understanding of requirements quality. Though recent publications make an effort at disentangling the complex concept of quality, the requirements quality…
▽ More
High-quality requirements minimize the risk of propagating defects to later stages of the software development life cycle. Achieving a sufficient level of quality is a major goal of requirements engineering. This requires a clear definition and understanding of requirements quality. Though recent publications make an effort at disentangling the complex concept of quality, the requirements quality research community lacks identity and clear structure which guides advances and puts new findings into an holistic perspective. In this research commentary we contribute (1) a harmonized requirements quality theory organizing its core concepts, (2) an evaluation of the current state of requirements quality research, and (3) a research roadmap to guide advancements in the field. We show that requirements quality research focuses on normative rules and mostly fails to connect requirements quality to its impact on subsequent software development activities, impeding the relevance of the research. Adherence to the proposed requirements quality theory and following the outlined roadmap will be a step towards amending this gap.
△ Less
Submitted 19 September, 2023;
originally announced September 2023.
-
A Field Test of Bandit Algorithms for Recommendations: Understanding the Validity of Assumptions on Human Preferences in Multi-armed Bandits
Authors:
Liu Leqi,
Giulio Zhou,
Fatma Kılınç-Karzan,
Zachary C. Lipton,
Alan L. Montgomery
Abstract:
Personalized recommender systems suffuse modern life, shaping what media we read and what products we consume. Algorithms powering such systems tend to consist of supervised learning-based heuristics, such as latent factor models with a variety of heuristically chosen prediction targets. Meanwhile, theoretical treatments of recommendation frequently address the decision-theoretic nature of the pro…
▽ More
Personalized recommender systems suffuse modern life, shaping what media we read and what products we consume. Algorithms powering such systems tend to consist of supervised learning-based heuristics, such as latent factor models with a variety of heuristically chosen prediction targets. Meanwhile, theoretical treatments of recommendation frequently address the decision-theoretic nature of the problem, including the need to balance exploration and exploitation, via the multi-armed bandits (MABs) framework. However, MAB-based approaches rely heavily on assumptions about human preferences. These preference assumptions are seldom tested using human subject studies, partly due to the lack of publicly available toolkits to conduct such studies. In this work, we conduct a study with crowdworkers in a comics recommendation MABs setting. Each arm represents a comic category, and users provide feedback after each recommendation. We check the validity of core MABs assumptions-that human preferences (reward distributions) are fixed over time-and find that they do not hold. This finding suggests that any MAB algorithm used for recommender systems should account for human preference dynamics. While answering these questions, we provide a flexible experimental framework for understanding human preference dynamics and testing MABs algorithms with human users. The code for our experimental framework and the collected data can be found at https://github.com/HumainLab/human-bandit-evaluation.
△ Less
Submitted 16 April, 2023;
originally announced April 2023.
-
Let's Stop Building at the Feet of Giants: Recovering unavailable Requirements Quality Artifacts
Authors:
Julian Frattini,
Lloyd Montgomery,
Davide Fucci,
Jannik Fischbach,
Michael Unterkalmsteiner,
Daniel Mendez
Abstract:
Requirements quality literature abounds with publications presenting artifacts, such as data sets and tools. However, recent systematic studies show that more than 80% of these artifacts have become unavailable or were never made public, limiting reproducibility and reusability. In this work, we report on an attempt to recover those artifacts. To that end, we requested corresponding authors of una…
▽ More
Requirements quality literature abounds with publications presenting artifacts, such as data sets and tools. However, recent systematic studies show that more than 80% of these artifacts have become unavailable or were never made public, limiting reproducibility and reusability. In this work, we report on an attempt to recover those artifacts. To that end, we requested corresponding authors of unavailable artifacts to recover and disclose them according to open science principles. Our results, based on 19 answers from 35 authors (54% response rate), include an assessment of the availability of requirements quality artifacts and a breakdown of authors' reasons for their continued unavailability. Overall, we improved the availability of seven data sets and seven implementations.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
-
An Exploratory Study of Documentation Strategies for Product Features in Popular GitHub Projects
Authors:
Tim Puhlfürß,
Lloyd Montgomery,
Walid Maalej
Abstract:
[Background] In large open-source software projects, development knowledge is often fragmented across multiple artefacts and contributors such that individual stakeholders are generally unaware of the full breadth of the product features. However, users want to know what the software is capable of, while contributors need to know where to fix, update, and add features. [Objective] This work aims a…
▽ More
[Background] In large open-source software projects, development knowledge is often fragmented across multiple artefacts and contributors such that individual stakeholders are generally unaware of the full breadth of the product features. However, users want to know what the software is capable of, while contributors need to know where to fix, update, and add features. [Objective] This work aims at understanding how feature knowledge is documented in GitHub projects and how it is linked (if at all) to the source code. [Method] We conducted an in-depth qualitative exploratory content analysis of 25 popular GitHub repositories that provided the documentation artefacts recommended by GitHub's Community Standards indicator. We first extracted strategies used to document software features in textual artefacts and then strategies used to link the feature documentation with source code. [Results] We observed feature documentation in all studied projects in artefacts such as READMEs, wikis, and website resource files. However, the features were often described in an unstructured way. Additionally, tracing techniques to connect feature documentation and source code were rarely used. [Conclusions] Our results suggest a lacking (or a low-prioritised) feature documentation in open-source projects, little use of normalised structures, and a rare explicit referencing to source code. As a result, product feature traceability is likely to be very limited, and maintainability to suffer over time.
△ Less
Submitted 2 August, 2022;
originally announced August 2022.
-
A Live Extensible Ontology of Quality Factors for Textual Requirements
Authors:
Julian Frattini,
Lloyd Montgomery,
Jannik Fischbach,
Michael Unterkalmsteiner,
Daniel Mendez,
Davide Fucci
Abstract:
Quality factors like passive voice or sentence length are commonly used in research and practice to evaluate the quality of natural language requirements since they indicate defects in requirements artifacts that potentially propagate to later stages in the development life cycle. However, as a research community, we still lack a holistic perspective on quality factors. This inhibits not only a co…
▽ More
Quality factors like passive voice or sentence length are commonly used in research and practice to evaluate the quality of natural language requirements since they indicate defects in requirements artifacts that potentially propagate to later stages in the development life cycle. However, as a research community, we still lack a holistic perspective on quality factors. This inhibits not only a comprehensive understanding of the existing body of knowledge but also the effective use and evolution of these factors. To this end, we propose an ontology of quality factors for textual requirements, which includes (1) a structure framing quality factors and related elements and (2) a central repository and web interface making these factors publicly accessible and usable. We contribute the first version of both by applying a rigorous ontology development method to 105 eligible primary studies and construct a first version of the repository and interface. We illustrate the usability of the ontology and invite fellow researchers to a joint community effort to complete and maintain this knowledge repository. We envision our ontology to reflect the community's harmonized perception of requirements quality factors, guide reporting of new quality factors, and provide central access to the current body of knowledge.
△ Less
Submitted 7 January, 2025; v1 submitted 13 June, 2022;
originally announced June 2022.
-
An Alternative Issue Tracking Dataset of Public Jira Repositories
Authors:
Lloyd Montgomery,
Clara Lüders,
Walid Maalej
Abstract:
Organisations use issue tracking systems (ITSs) to track and document their projects' work in units called issues. This style of documentation encourages evolutionary refinement, as each issue can be independently improved, commented on, linked to other issues, and progressed through the organisational workflow. Commonly studied ITSs so far include GitHub, GitLab, and Bugzilla, while Jira, one of…
▽ More
Organisations use issue tracking systems (ITSs) to track and document their projects' work in units called issues. This style of documentation encourages evolutionary refinement, as each issue can be independently improved, commented on, linked to other issues, and progressed through the organisational workflow. Commonly studied ITSs so far include GitHub, GitLab, and Bugzilla, while Jira, one of the most popular ITS in practice with a wealth of additional information, has yet to receive similar attention. Unfortunately, diverse public Jira datasets are rare, likely due to the difficulty in finding and accessing these repositories. With this paper, we release a dataset of 16 public Jiras with 1822 projects, spanning 2.7 million issues with a combined total of 32 million changes, 9 million comments, and 1 million issue links. We believe this Jira dataset will lead to many fruitful research projects investigating issue evolution, issue linking, cross-project analysis, as well as cross-tool analysis when combined with existing well-studied ITS datasets.
△ Less
Submitted 25 March, 2022; v1 submitted 20 January, 2022;
originally announced January 2022.
-
Rebounding Bandits for Modeling Satiation Effects
Authors:
Liu Leqi,
Fatma Kilinc-Karzan,
Zachary C. Lipton,
Alan L. Montgomery
Abstract:
Psychological research shows that enjoyment of many goods is subject to satiation, with short-term satisfaction declining after repeated exposures to the same item. Nevertheless, proposed algorithms for powering recommender systems seldom model these dynamics, instead proceeding as though user preferences were fixed in time. In this work, we introduce rebounding bandits, a multi-armed bandit setup…
▽ More
Psychological research shows that enjoyment of many goods is subject to satiation, with short-term satisfaction declining after repeated exposures to the same item. Nevertheless, proposed algorithms for powering recommender systems seldom model these dynamics, instead proceeding as though user preferences were fixed in time. In this work, we introduce rebounding bandits, a multi-armed bandit setup, where satiation dynamics are modeled as time-invariant linear dynamical systems. Expected rewards for each arm decline monotonically with consecutive exposures to it and rebound towards the initial reward whenever that arm is not pulled. Unlike classical bandit settings, methods for tackling rebounding bandits must plan ahead and model-based methods rely on estimating the parameters of the satiation dynamics. We characterize the planning problem, showing that the greedy policy is optimal when the arms exhibit identical deterministic dynamics. To address stochastic satiation dynamics with unknown parameters, we propose Explore-Estimate-Plan (EEP), an algorithm that pulls arms methodically, estimates the system dynamics, and then plans accordingly.
△ Less
Submitted 27 October, 2021; v1 submitted 12 November, 2020;
originally announced November 2020.
-
Renovating Requirements Engineering: First Thoughts to Shape Requirements Engineering as a Profession
Authors:
Yen Dieu Pham,
Lloyd Montgomery,
Walid Maalej
Abstract:
Legacy software systems typically include vital data for organizations that use them and should thus to be regularly maintained. Ideally, organizations should rely on Requirements Engineers to understand and manage changes of stakeholder needs and system constraints. However, due to time and cost pressure, and with a heavy focus on implementation, organizations often choose to forgo Requirements E…
▽ More
Legacy software systems typically include vital data for organizations that use them and should thus to be regularly maintained. Ideally, organizations should rely on Requirements Engineers to understand and manage changes of stakeholder needs and system constraints. However, due to time and cost pressure, and with a heavy focus on implementation, organizations often choose to forgo Requirements Engineers and rather focus on ad-hoc bug fixing and maintenance. This position paper discusses what Requirements Engineers could possibly learn from other similar roles to become crucial for the evolution of legacy systems. Particularly, we compare the roles of Requirements Engineers (according to IREB), Building Architects (according to the German regulations), and Product Owners (according to "The Scrum-Guide"). We discuss overlaps along four dimensions: liability, self-portrayal, core activities, and artifacts. Finally we draw insights from these related fields to foster the concept of a Requirements Engineer as a distinguished profession.
△ Less
Submitted 26 October, 2020;
originally announced October 2020.
-
How angry are your customers? Sentiment analysis of support tickets that escalate
Authors:
Colin Werner,
Lloyd Montgomery,
Sanja Dodos,
Gabriel Tapuc,
Diksha Sharma,
Daniela Damian
Abstract:
Software support ticket escalations can be an extremely costly burden for software organizations all over the world. Consequently, there exists an interest in researching how to better enable support analysts to handle such escalations. In order to do so, we need to develop tools to reliably predict if, and when, a support ticket becomes a candidate for escalation. This paper explores the use of s…
▽ More
Software support ticket escalations can be an extremely costly burden for software organizations all over the world. Consequently, there exists an interest in researching how to better enable support analysts to handle such escalations. In order to do so, we need to develop tools to reliably predict if, and when, a support ticket becomes a candidate for escalation. This paper explores the use of sentiment analysis tools on customer-support analyst conversations to find indicators of when a particular support ticket may be escalated. The results of this research indicate a considerable difference in the sentiment between escalated support tickets and non-escalated support tickets. Thus, this preliminary research provides us with the necessary information to further investigate how we can reliably predict support ticket escalations, and subsequently to provide insight to support analysts to better enable them to handle support tickets that may be escalated.
△ Less
Submitted 26 October, 2020;
originally announced October 2020.
-
Escalation Prediction using Feature Engineering: Addressing Support Ticket Escalations within IBM's Ecosystem
Authors:
Lloyd Montgomery
Abstract:
Large software organizations handle many customer support issues every day in the form of bug reports, feature requests, and general misunderstandings as submitted by customers. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit thei…
▽ More
Large software organizations handle many customer support issues every day in the form of bug reports, feature requests, and general misunderstandings as submitted by customers. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. Whenever insufficient attention is given to support issues, there is a chance customers will escalate their issues, and escalation to management is time-consuming and expensive, especially for large organizations managing hundreds of customers and thousands of support tickets. This thesis provides a step towards simplifying the job for support analysts and managers, particularly in predicting the risk of escalating support tickets. In a field study at our large industrial partner, IBM, a design science methodology was employed to characterize the support process and data available to IBM analysts in managing escalations. Through iterative cycles of design and evaluation, support analysts' expert knowledge about their customers was translated into features of a support ticket model to be implemented into a Machine Learning model to predict support ticket escalations. The Machine Learning model was trained and evaluated on over 2.5 million support tickets and 10,000 escalations, obtaining a recall of 79.9% and an 80.8% reduction in the workload for support analysts looking to identify support tickets at risk of escalation. The features developed in the Support Ticket Model are designed to serve as a starting place for organizations interested in implementing the model to predict support ticket escalations, and for future researchers to build on to advance research in Escalation Prediction.
△ Less
Submitted 12 October, 2020;
originally announced October 2020.
-
Customer Support Ticket Escalation Prediction using Feature Engineering
Authors:
Lloyd Montgomery,
Daniela Damian,
Tyson Bulmer,
Shaikh Quader
Abstract:
Understanding and keeping the customer happy is a central tenet of requirements engineering. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. If insufficient attention is given to su…
▽ More
Understanding and keeping the customer happy is a central tenet of requirements engineering. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. If insufficient attention is given to support issues, however, their escalation to management becomes time-consuming and expensive, especially for large organizations managing hundreds of customers and thousands of support tickets. Our work provides a step towards simplifying the job of support analysts and managers, particularly in predicting the risk of escalating support tickets. In a field study at our large industrial partner, IBM, we used a design science research methodology to characterize the support process and data available to IBM analysts in managing escalations. We then implemented these features into a machine learning model to predict support ticket escalations. We trained and evaluated our machine learning model on over 2.5 million support tickets and 10,000 escalations, obtaining a recall of 87.36% and an 88.23% reduction in the workload for support analysts looking to identify support tickets at risk of escalation. Finally, in addition to these research evaluation activities, we compared the performance of our support ticket model with that of a model developed with no feature engineering; the support ticket model features outperformed the non-engineered model. The artifacts created in this research are designed to serve as a starting place for organizations interested in predicting support ticket escalations, and for future researchers to build on to advance research in escalation prediction.
△ Less
Submitted 10 October, 2020;
originally announced October 2020.
-
Predicting Developers' IDE Commands with Machine Learning
Authors:
Tyson Bulmer,
Lloyd Montgomery,
Daniela Damian
Abstract:
When a developer is writing code they are usually focused and in a state-of-mind which some refer to as flow. Breaking out of this flow can cause the developer to lose their train of thought and have to start their thought process from the beginning. This loss of thought can be caused by interruptions and sometimes slow IDE interactions. Predictive functionality has been harnessed in user applicat…
▽ More
When a developer is writing code they are usually focused and in a state-of-mind which some refer to as flow. Breaking out of this flow can cause the developer to lose their train of thought and have to start their thought process from the beginning. This loss of thought can be caused by interruptions and sometimes slow IDE interactions. Predictive functionality has been harnessed in user applications to speed up load times, such as in Google Chrome's browser which has a feature called "Predicting Network Actions". This will pre-load web-pages that the user is most likely to click through. This mitigates the interruption that load times can introduce. In this paper we seek to make the first step towards predicting user commands in the IDE. Using the MSR 2018 Challenge Data of over 3000 developer session and over 10 million recorded events, we analyze and cleanse the data to be parsed into event series, which can then be used to train a variety of machine learning models, including a neural network, to predict user induced commands. Our highest performing model is able to obtain a 5 cross-fold validation prediction accuracy of 64%.
△ Less
Submitted 10 October, 2020;
originally announced October 2020.
-
ECrits - Visualizing Support Ticket Escalation Risk
Authors:
Lloyd Montgomery,
Emma Reading,
Daniela Damian
Abstract:
Managing support tickets in large, multi-product organizations is difficult. Failure to meet the expectations of customers can lead to the escalation of support tickets, which is costly for IBM in terms of customer relationships and resources spent addressing the escalation. Keeping the customer happy is an important task in requirements engineering, which often comes in the form of handling their…
▽ More
Managing support tickets in large, multi-product organizations is difficult. Failure to meet the expectations of customers can lead to the escalation of support tickets, which is costly for IBM in terms of customer relationships and resources spent addressing the escalation. Keeping the customer happy is an important task in requirements engineering, which often comes in the form of handling their problems brought forth in support tickets. Proper attention to customers, their issues, and the bottom-up requirements that surface through bug reports can be difficult when the support process involves spending a lot of time managing customers to prevent escalations. For any given support analyst, understanding the customer is achievable through time spent looking through past and present support tickets within their organization; however, this solution does not scale up to account for all support tickets across all product teams. ECrits is a tool developed to help mitigate information overload by selectively mining customer information from support ticket repositories, displaying that data to support analysts, and doing predictive modelling on that data to suggest which support tickets are likely to escalate.
△ Less
Submitted 4 January, 2019;
originally announced January 2019.
-
What do Support Analysts Know about Their Customers? On the Study and Prediction of Support Ticket Escalations in Large Software Organizations
Authors:
Lloyd Montgomery,
Daniela Damian
Abstract:
Understanding and keeping the customer happy is a central tenet of requirements engineering. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. Whenever insufficient attention is given…
▽ More
Understanding and keeping the customer happy is a central tenet of requirements engineering. Strategies to gather, analyze, and negotiate requirements are complemented by efforts to manage customer input after products have been deployed. For the latter, support tickets are key in allowing customers to submit their issues, bug reports, and feature requests. Whenever insufficient attention is given to support issues, however, their escalation to management is time-consuming and expensive, especially for large organizations managing hundreds of customers and thousands of support tickets. Our work provides a step towards simplifying the job of support analysts and managers, particularly in predicting the risk of escalating support tickets. In a field study at our large industrial partner, IBM, we used a design science methodology to characterize the support process and data available to IBM analysts in managing escalations. Through iterative cycles of design and evaluation, we translated our understanding of support analysts' expert knowledge of their customers into features of a support ticket model to be implemented into a Machine Learning model to predict support ticket escalations. We trained and evaluated our Machine Learning model on over 2.5 million support tickets and 10,000 escalations, obtaining a recall of 79.9% and an 80.8% reduction in the workload for support analysts looking to identify support tickets at risk of escalation. Further on-site evaluations, through a prototype tool we developed to implement our Machine Learning techniques in practice, showed more efficient weekly support-ticket-management meetings. The features we developed in the Support Ticket Model are designed to serve as a starting place for organizations interested in implementing our model to predict support ticket escalations, and for future researchers to build on to advance research in ...
△ Less
Submitted 4 January, 2019;
originally announced January 2019.
-
Towards a Live Anonymous Question Queue To Address Student Apprehension
Authors:
Lloyd Montgomery,
Guy Evans,
Francis Harrison,
Daniela Damian
Abstract:
In today's university climate many first and second year classes have over a hundred students. Large classrooms make some students apprehensive about asking questions. An anonymous method of submitting questions to an instructor would allow students to ask their questions without feeling apprehensive. In this paper we propose a Live Anonymous Question Queue (LAQQ), a system that facilitates anonym…
▽ More
In today's university climate many first and second year classes have over a hundred students. Large classrooms make some students apprehensive about asking questions. An anonymous method of submitting questions to an instructor would allow students to ask their questions without feeling apprehensive. In this paper we propose a Live Anonymous Question Queue (LAQQ), a system that facilitates anonymous question submissions in real time to mitigate student apprehension, increase student participation, and provide real-time feedback to the instructor. To study the necessary features of an LAQQ, we conducted a study of a system, namely Google Moderator, which best approached our concept of an LAQQ. We deployed Google moderator in large lectures and studied its support of a number of features that we envisioned for an LAQQ. Through our class observations, interviews with instructors, and surveys with the students, our results suggest that an LAQQ system must provide support for: notification of question submission to provide awareness for the instructor, and context for questions to allow an instructor to easily answer a question. Additionally our results suggest that an LAQQ system must be accessible and usable on multiple platforms. Finally our results suggest that in order to be successful in the classroom an LAQQ system must be fully adopted by the instructor and the classroom organizational structure must change to accommodate the use of the LAQQ.
△ Less
Submitted 4 January, 2019;
originally announced January 2019.
-
A Simple NLP-based Approach to Support Onboarding and Retention in Open Source Communities
Authors:
Christoph Stanik,
Lloyd Montgomery,
Daniel Martens,
Davide Fucci,
Walid Maalej
Abstract:
Successful open source communities are constantly looking for new members and helping them become active developers. A common approach for developer onboarding in open source projects is to let newcomers focus on relevant yet easy-to-solve issues to familiarize themselves with the code and the community. The goal of this research is twofold. First, we aim at automatically identifying issues that n…
▽ More
Successful open source communities are constantly looking for new members and helping them become active developers. A common approach for developer onboarding in open source projects is to let newcomers focus on relevant yet easy-to-solve issues to familiarize themselves with the code and the community. The goal of this research is twofold. First, we aim at automatically identifying issues that newcomers can resolve by analyzing the history of resolved issues by simply using the title and description of issues. Second, we aim at automatically identifying issues, that can be resolved by newcomers who later become active developers. We mined the issue trackers of three large open source projects and extracted natural language features from the title and description of resolved issues. In a series of experiments, we optimized and compared the accuracy of four supervised classifiers to address our research goals. Random Forest, achieved up to 91% precision (F1-score 72%) towards the first goal while for the second goal, Decision Tree achieved a precision of 92% (F1-score 91%). A qualitative evaluation gave insights on what information in the issue description is helpful for newcomers. Our approach can be used to automatically identify, label, and recommend issues for newcomers in open source software projects based only on the text of the issues.
△ Less
Submitted 16 August, 2018; v1 submitted 7 June, 2018;
originally announced June 2018.
-
Finding ECM-friendly curves through a study of Galois properties
Authors:
Razvan Barbulescu,
Joppe W. Bos,
Cyril Bouvier,
Thorsten Kleinjung,
Peter L. Montgomery
Abstract:
In this paper we prove some divisibility properties of the cardinality of elliptic curves modulo primes. These proofs explain the good behavior of certain parameters when using Montgomery or Edwards curves in the setting of the elliptic curve method (ECM) for integer factorization. The ideas of the proofs help us to find new families of elliptic curves with good division properties which increase…
▽ More
In this paper we prove some divisibility properties of the cardinality of elliptic curves modulo primes. These proofs explain the good behavior of certain parameters when using Montgomery or Edwards curves in the setting of the elliptic curve method (ECM) for integer factorization. The ideas of the proofs help us to find new families of elliptic curves with good division properties which increase the success probability of ECM.
△ Less
Submitted 4 September, 2012; v1 submitted 20 February, 2012;
originally announced February 2012.
-
Factorizations of Cunningham numbers with bases 13 to 99
Authors:
Richard P. Brent,
Peter L. Montgomery,
Herman J. J. te Riele
Abstract:
This Report updates the tables of factorizations of a^n +- 1 for 13 < a < 100, previously published as CWI Report NM-R9212 (June 1992) and updated in CWI Report NM-R9419 (Update 1, September 1994) and CWI Report NM-R9609 (Update 2, March 1996). A total of 951 new entries in the tables are given here. The factorizations are now complete for n < 76, and there are no composite cofactors smaller than…
▽ More
This Report updates the tables of factorizations of a^n +- 1 for 13 < a < 100, previously published as CWI Report NM-R9212 (June 1992) and updated in CWI Report NM-R9419 (Update 1, September 1994) and CWI Report NM-R9609 (Update 2, March 1996). A total of 951 new entries in the tables are given here. The factorizations are now complete for n < 76, and there are no composite cofactors smaller than 10^102.
△ Less
Submitted 19 April, 2010; v1 submitted 19 April, 2010;
originally announced April 2010.
