-
Quantum Testing in the Wild: A Case Study with Qiskit Algorithms
Authors:
Neilson Carlos Leite Ramalho,
Erico Augusto da Silva,
Higor Amario de Souza,
Marcos Lordello Chaim
Abstract:
Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation.…
▽ More
Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation. With the increasing interest in the field, there are challenges and opportunities for academics and practitioners in terms of software engineering practices, particularly in testing quantum programs. This paper presents an empirical study of testing patterns in quantum algorithms. We analyzed all the tests handling quantum aspects of the implementations in the Qiskit Algorithms library and identified seven distinct patterns that make use of (1) fixed seeds for algorithms based on random elements; (2) deterministic oracles; (3) precise and approximate assertions; (4) Data-Driven Testing (DDT); (5) functional testing; (6) testing for intermediate parts of the algorithms being tested; and (7) equivalence checking for quantum circuits. Our results show a prevalence of classical testing techniques to test the quantum-related elements of the library, while recent advances from the research community have yet to achieve wide adoption among practitioners.
△ Less
Submitted 11 January, 2025;
originally announced January 2025.
-
Testing and Debugging Quantum Programs: The Road to 2030
Authors:
Neilson Carlos Leite Ramalho,
Higor Amario de Souza,
Marcos Lordello Chaim
Abstract:
Quantum computing has existed in the theoretical realm for several decades. Recently, quantum computing has re-emerged as a promising technology to solve problems that a classical computer could take hundreds of years to solve. However, there are challenges and opportunities for academics and practitioners regarding software engineering practices for testing and debugging quantum programs. This pa…
▽ More
Quantum computing has existed in the theoretical realm for several decades. Recently, quantum computing has re-emerged as a promising technology to solve problems that a classical computer could take hundreds of years to solve. However, there are challenges and opportunities for academics and practitioners regarding software engineering practices for testing and debugging quantum programs. This paper presents a roadmap for addressing these challenges, pointing out the existing gaps in the literature and suggesting research directions. We discuss the limitations caused by noise, the no-cloning theorem, the lack of a standard architecture for quantum computers, among others. Regarding testing, we highlight gaps and opportunities related to transpilation, mutation analysis, input states with hybrid interfaces, program analysis, and coverage. For debugging, we present the current strategies, including classical techniques applied to quantum programs, quantum-specific assertions, and quantum-related bug patterns. We introduce a conceptual model to illustrate concepts regarding the testing and debugging of quantum programs and the relationship between them. Those concepts are used to identify and discuss research challenges to cope with quantum programs through 2030, focusing on the interfaces between classical and quantum computing and on creating testing and debugging techniques that take advantage of the unique quantum computing characteristics.
△ Less
Submitted 11 October, 2024; v1 submitted 15 May, 2024;
originally announced May 2024.
-
A Data Flow Analysis Framework for Data Flow Subsumption
Authors:
Marcos Lordello Chaim,
Kesina Baral,
Jeff Offutt
Abstract:
Data flow testing creates test requirements as definition-use (DU) associations, where a definition is a program location that assigns a value to a variable and a use is a location where that value is accessed. Data flow testing is expensive, largely because of the number of test requirements. Luckily, many DU-associations are redundant in the sense that if one test requirement (e.g., node, edge,…
▽ More
Data flow testing creates test requirements as definition-use (DU) associations, where a definition is a program location that assigns a value to a variable and a use is a location where that value is accessed. Data flow testing is expensive, largely because of the number of test requirements. Luckily, many DU-associations are redundant in the sense that if one test requirement (e.g., node, edge, DU-association) is covered, other DU-associations are guaranteed to also be covered. This relationship is called subsumption. Thus, testers can save resources by only covering DU-associations that are not subsumed by other testing requirements. In this work, we formally describe the Data Flow Subsumption Framework (DSF) conceived to tackle the data flow subsumption problem. We show that DFS is a distributive data flow analysis framework which allows efficient iterative algorithms to find the Meet-Over-All-Paths (MOP) solution for DSF transfer functions. The MOP solution implies that the results at a point $p$ are valid for all paths that reach $p$. We also present an algorithm, called Subsumption Algorithm (SA), that uses DSF transfer functions and iterative algorithms to find the local DU-associations-node subsumption; that is, the set of DU-associations that are covered whenever a node $n$ is toured by a test. A proof of SA's correctness is presented and its complexity is analyzed.
△ Less
Submitted 14 January, 2021;
originally announced January 2021.
-
Evaluating data-flow coverage in spectrum-based fault localization
Authors:
Henrique Lemos Ribeiro,
Higor Amario de Souza,
Roberto Paulo de Andrioli Araujo,
Marcos Lordello Chaim,
Fabio Kon
Abstract:
Background: Debugging is a key task during the software development cycle. Spectrum-based Fault Localization (SFL) is a promising technique to improve and automate debugging. SFL techniques use control-flow spectra to pinpoint the most suspicious program elements. However, data-flow spectra provide more detailed information about the program execution, which may be useful for fault localization. A…
▽ More
Background: Debugging is a key task during the software development cycle. Spectrum-based Fault Localization (SFL) is a promising technique to improve and automate debugging. SFL techniques use control-flow spectra to pinpoint the most suspicious program elements. However, data-flow spectra provide more detailed information about the program execution, which may be useful for fault localization. Aims: We evaluate the effectiveness and efficiency of ten SFL ranking metrics using data-flow spectra. Method: We compare the performance of data- and control-flow spectra for SFL using 163 faults from 5 real-world open source programs, which contain from 468 to 4130 test cases. The data- and control-flow spectra types used in our evaluation are definition-use associations (DUAs) and lines, respectively. Results: Using data-flow spectra, up to 50% more faults are ranked in the top-15 positions compared to control-flow spectra. Also, most SFL ranking metrics present better effectiveness using data-flow to inspect up to the top-40 positions. The execution cost of data-flow spectra is higher than control-flow, taking from 22 seconds to less than 9 minutes. Data-flow has an average overhead of 353% for all programs, while the average overhead for control-flow is of 102%. Conclusions: The results suggest that SFL techniques can benefit from using data-flow spectra to classify faults in better positions, which may lead developers to inspect less code to find bugs. The execution cost to gather data-flow is higher compared to control-flow, but it is not prohibitive. Moreover, data-flow spectra also provide information about suspicious variables for fault localization, which may improve the developers' performance using SFL.
△ Less
Submitted 27 June, 2019;
originally announced June 2019.
-
Spectrum-based Software Fault Localization: A Survey of Techniques, Advances, and Challenges
Authors:
Higor A. de Souza,
Marcos L. Chaim,
Fabio Kon
Abstract:
Despite being one of the most basic tasks in software development, debugging is still performed in a mostly manual way, leading to high cost and low performance. To address this problem, researchers have studied promising approaches, such as Spectrum-based Fault Localization (SFL) techniques, which pinpoint program elements more likely to contain faults. This survey discusses the state-of-the-art…
▽ More
Despite being one of the most basic tasks in software development, debugging is still performed in a mostly manual way, leading to high cost and low performance. To address this problem, researchers have studied promising approaches, such as Spectrum-based Fault Localization (SFL) techniques, which pinpoint program elements more likely to contain faults. This survey discusses the state-of-the-art of SFL, including the different techniques that have been proposed, the type and number of faults they address, the types of spectra they use, the programs they utilize in their validation, the testing data that support them, and their use at industrial settings. Notwithstanding the advances, there are still challenges for the industry to adopt these techniques, which we analyze in this paper. SFL techniques should propose new ways to generate reduced sets of suspicious entities, combine different spectra to fine-tune the fault localization ability, use strategies to collect fine-grained coverage levels from suspicious coarser levels for balancing execution costs and output precision, and propose new techniques to cope with multiple-fault programs. Moreover, additional user studies are needed to understand better how SFL techniques can be used in practice. We conclude by presenting a concept map about topics and challenges for future research in SFL.
△ Less
Submitted 26 November, 2017; v1 submitted 14 July, 2016;
originally announced July 2016.
