-
Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures
Authors:
Owain Parry,
Gregory Kapfhammer,
Michael Hilton,
Phil McMinn
Abstract:
Flaky tests produce inconsistent outcomes without code changes, creating major challenges for software developers. An industrial case study reported that developers spend 1.28% of their time repairing flaky tests at a monthly cost of $2,250. We discovered that flaky tests often exist in clusters, with co-occurring failures that share the same root causes, which we call systemic flakiness. This sug…
▽ More
Flaky tests produce inconsistent outcomes without code changes, creating major challenges for software developers. An industrial case study reported that developers spend 1.28% of their time repairing flaky tests at a monthly cost of $2,250. We discovered that flaky tests often exist in clusters, with co-occurring failures that share the same root causes, which we call systemic flakiness. This suggests that developers can reduce repair costs by addressing shared root causes, enabling them to fix multiple flaky tests at once rather than tackling them individually. This study represents an inflection point by challenging the deep-seated assumption that flaky test failures are isolated occurrences. We used an established dataset of 10,000 test suite runs from 24 Java projects on GitHub, spanning domains from data orchestration to job scheduling. It contains 810 flaky tests, which we levered to perform a mixed-method empirical analysis of co-occurring flaky test failures. Systemic flakiness is significant and widespread. We performed agglomerative clustering of flaky tests based on their failure co-occurrence, finding that 75% of flaky tests across all projects belong to a cluster, with a mean cluster size of 13.5 flaky tests. Instead of requiring 10,000 test suite runs to identify systemic flakiness, we demonstrated a lightweight alternative by training machine learning models based on static test case distance measures. Through manual inspection of stack traces, conducted independently by four authors and resolved through negotiated agreement, we identified intermittent networking issues and instabilities in external dependencies as the predominant causes of systemic flakiness.
△ Less
Submitted 23 April, 2025;
originally announced April 2025.
-
QAOA-PCA: Enhancing Efficiency in the Quantum Approximate Optimization Algorithm via Principal Component Analysis
Authors:
Owain Parry,
Phil McMinn
Abstract:
The Quantum Approximate Optimization Algorithm (QAOA) is a promising variational algorithm for solving combinatorial optimization problems on near-term devices. However, as the number of layers in a QAOA circuit increases, which is correlated with the quality of the solution, the number of parameters to optimize grows linearly. This results in more iterations required by the classical optimizer, w…
▽ More
The Quantum Approximate Optimization Algorithm (QAOA) is a promising variational algorithm for solving combinatorial optimization problems on near-term devices. However, as the number of layers in a QAOA circuit increases, which is correlated with the quality of the solution, the number of parameters to optimize grows linearly. This results in more iterations required by the classical optimizer, which results in an increasing computational burden as more circuit executions are needed. To mitigate this issue, we introduce QAOA-PCA, a novel reparameterization technique that employs Principal Component Analysis (PCA) to reduce the dimensionality of the QAOA parameter space. By extracting principal components from optimized parameters of smaller problem instances, QAOA-PCA facilitates efficient optimization with fewer parameters on larger instances. Our empirical evaluation on the prominent MaxCut problem demonstrates that QAOA-PCA consistently requires fewer iterations than standard QAOA, achieving substantial efficiency gains. While this comes at the cost of a slight reduction in approximation ratio compared to QAOA with the same number of layers, QAOA-PCA almost always outperforms standard QAOA when matched by parameter count. QAOA-PCA strikes a favorable balance between efficiency and performance, reducing optimization overhead without significantly compromising solution quality.
△ Less
Submitted 23 April, 2025;
originally announced April 2025.
-
Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation
Authors:
Islam T. Elgendy,
Robert M. Hierons,
Phil McMinn
Abstract:
Regression testing assures software correctness after changes but is resource-intensive. Test Case Prioritisation (TCP) mitigates this by ordering tests to maximise early fault detection. Diversity-based TCP prioritises dissimilar tests, assuming they exercise different system parts and uncover more faults. Traditional static diversity-based TCP approaches (i.e., methods that utilise the dissimila…
▽ More
Regression testing assures software correctness after changes but is resource-intensive. Test Case Prioritisation (TCP) mitigates this by ordering tests to maximise early fault detection. Diversity-based TCP prioritises dissimilar tests, assuming they exercise different system parts and uncover more faults. Traditional static diversity-based TCP approaches (i.e., methods that utilise the dissimilarity of tests), like the state-of-the-art FAST approach, rely on textual diversity from test source code, which is effective but inefficient due to its relative verbosity and redundancies affecting similarity calculations. This paper is the first to study bytecode as the basis of diversity in TCP, leveraging its compactness for improved efficiency and accuracy. An empirical study on seven Defects4J projects shows that bytecode diversity improves fault detection by 2.3-7.8% over text-based TCP. It is also 2-3 orders of magnitude faster in one TCP approach and 2.5-6 times faster in FAST-based TCP. Filtering specific bytecode instructions improves efficiency up to fourfold while maintaining effectiveness, making bytecode diversity a superior static approach.
△ Less
Submitted 17 April, 2025;
originally announced April 2025.
-
Improving the Reliability of Quantum Circuits by Evolving Heterogeneous Ensembles
Authors:
Owain Parry,
John Clark,
Phil McMinn
Abstract:
Quantum computers can perform certain operations exponentially faster than classical computers, but designing quantum circuits is challenging. To that end, researchers used evolutionary algorithms to produce probabilistic quantum circuits that give the correct output more often than not for any input. They can be executed multiple times, with the outputs combined using a classical method (such as…
▽ More
Quantum computers can perform certain operations exponentially faster than classical computers, but designing quantum circuits is challenging. To that end, researchers used evolutionary algorithms to produce probabilistic quantum circuits that give the correct output more often than not for any input. They can be executed multiple times, with the outputs combined using a classical method (such as voting) to produce the final output, effectively creating a homogeneous ensemble of circuits (i.e., all identical). Inspired by n-version programming and ensemble learning, we developed a tool that uses an evolutionary algorithm to generate heterogeneous ensembles of circuits (i.e., all different), named QuEEn. We used it to evolve ensembles to solve the Iris classification problem. When using ideal simulation, we found the performance of heterogeneous ensembles to be greater than that of homogeneous ensembles to a statistically significant degree. When using noisy simulation, we still observed a statistically significant improvement in the majority of cases. Our results indicate that evolving heterogeneous ensembles is an effective strategy for improving the reliability of quantum circuits. This is particularly relevant in the current NISQ era of quantum computing where computers do not yet have good tolerance to quantum noise.
△ Less
Submitted 13 September, 2024;
originally announced September 2024.
-
A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing
Authors:
Islam T. Elgendy,
Robert M. Hierons,
Phil McMinn
Abstract:
There has been a significant amount of interest regarding the use of diversity-based testing techniques in software testing over the past two decades. Diversity-based testing (DBT) technique uses similarity metrics to leverage the dissimilarity between software artefacts - such as requirements, abstract models, program structures, or inputs - in order to address a software testing problem. DBT tec…
▽ More
There has been a significant amount of interest regarding the use of diversity-based testing techniques in software testing over the past two decades. Diversity-based testing (DBT) technique uses similarity metrics to leverage the dissimilarity between software artefacts - such as requirements, abstract models, program structures, or inputs - in order to address a software testing problem. DBT techniques have been used to assist in finding solutions to several different types of problems including generating test cases, prioritising them, and reducing very large test suites. This paper is a systematic survey of DBT techniques that summarises the key aspects and trends of 144 papers that report the use of 70 different similarity metrics with 24 different types of software artefacts, which have been used by researchers to tackle 11 different types of software testing problems. We further present an analysis of the recent trends in DBT techniques and review the different application domains to which the techniques have been applied, giving an overview of the tools developed by researchers to do so. Finally, the paper identifies some DBT challenges that are potential topics for future work.
△ Less
Submitted 16 November, 2023;
originally announced November 2023.
-
Do Automatic Test Generation Tools Generate Flaky Tests?
Authors:
Martin Gruber,
Muhammad Firhard Roslan,
Owain Parry,
Fabian Scharnböck,
Phil McMinn,
Gordon Fraser
Abstract:
Non-deterministic test behavior, or flakiness, is common and dreaded among developers. Researchers have studied the issue and proposed approaches to mitigate it. However, the vast majority of previous work has only considered developer-written tests. The prevalence and nature of flaky tests produced by test generation tools remain largely unknown. We ask whether such tools also produce flaky tests…
▽ More
Non-deterministic test behavior, or flakiness, is common and dreaded among developers. Researchers have studied the issue and proposed approaches to mitigate it. However, the vast majority of previous work has only considered developer-written tests. The prevalence and nature of flaky tests produced by test generation tools remain largely unknown. We ask whether such tools also produce flaky tests and how these differ from developer-written ones. Furthermore, we evaluate mechanisms that suppress flaky test generation. We sample 6 356 projects written in Java or Python. For each project, we generate tests using EvoSuite (Java) and Pynguin (Python), and execute each test 200 times, looking for inconsistent outcomes. Our results show that flakiness is at least as common in generated tests as in developer-written tests. Nevertheless, existing flakiness suppression mechanisms implemented in EvoSuite are effective in alleviating this issue (71.7 % fewer flaky tests). Compared to developer-written flaky tests, the causes of generated flaky tests are distributed differently. Their non-deterministic behavior is more frequently caused by randomness, rather than by networking and concurrency. Using flakiness suppression, the remaining flaky tests differ significantly from any flakiness previously reported, where most are attributable to runtime optimizations and EvoSuite-internal resource thresholds. These insights, with the accompanying dataset, can help maintainers to improve test generation tools, give recommendations for developers using these tools, and serve as a foundation for future research in test flakiness or test generation.
△ Less
Submitted 8 October, 2023;
originally announced October 2023.
-
Gradeer: An Open-Source Modular Hybrid Grader
Authors:
Benjamin Clegg,
Maria-Cruz Villa-Uriol,
Phil McMinn,
Gordon Fraser
Abstract:
Automated assessment has been shown to greatly simplify the process of assessing students' programs. However, manual assessment still offers benefits to both students and tutors. We introduce Gradeer, a hybrid assessment tool, which allows tutors to leverage the advantages of both automated and manual assessment. The tool features a modular design, allowing new grading functionality to be added. G…
▽ More
Automated assessment has been shown to greatly simplify the process of assessing students' programs. However, manual assessment still offers benefits to both students and tutors. We introduce Gradeer, a hybrid assessment tool, which allows tutors to leverage the advantages of both automated and manual assessment. The tool features a modular design, allowing new grading functionality to be added. Gradeer directly assists manual grading, by automatically loading code inspectors, running students' programs, and allowing grading to be stopped and resumed in place at a later time. We used Gradeer to assess an end of year assignment for an introductory Java programming course, and found that its hybrid approach offers several benefits.
△ Less
Submitted 12 February, 2021;
originally announced February 2021.
