-
Duplicate Bug Report Detection: How Far Are We?
Authors:
Ting Zhang,
DongGyun Han,
Venkatesh Vinayakarao,
Ivana Clairine Irsan,
Bowen Xu,
Ferdian Thung,
David Lo,
Lingxiao Jiang
Abstract:
Many Duplicate Bug Report Detection (DBRD) techniques have been proposed in the research literature. The industry uses some other techniques. Unfortunately, there is insufficient comparison among them, and it is unclear how far we have been. This work fills this gap by comparing the aforementioned techniques. To compare them, we first need a benchmark that can estimate how a tool would perform if…
▽ More
Many Duplicate Bug Report Detection (DBRD) techniques have been proposed in the research literature. The industry uses some other techniques. Unfortunately, there is insufficient comparison among them, and it is unclear how far we have been. This work fills this gap by comparing the aforementioned techniques. To compare them, we first need a benchmark that can estimate how a tool would perform if applied in a realistic setting today. Thus, we first investigated potential biases that affect the fair comparison of the accuracy of DBRD techniques. Our experiments suggest that data age and issue tracking system choice cause a significant difference. Based on these findings, we prepared a new benchmark. We then used it to evaluate DBRD techniques to estimate better how far we have been. Surprisingly, a simpler technique outperforms recently proposed sophisticated techniques on most projects in our benchmark. In addition, we compared the DBRD techniques proposed in research with those used in Mozilla and VSCode. Surprisingly, we observe that a simple technique already adopted in practice can achieve comparable results as a recently proposed research tool. Our study gives reflections on the current state of DBRD, and we share our insights to benefit future DBRD research.
△ Less
Submitted 1 December, 2022;
originally announced December 2022.
-
Evaluating Commit Message Generation: To BLEU Or Not To BLEU?
Authors:
Samanta Dey,
Venkatesh Vinayakarao,
Monika Gupta,
Sampath Dechu
Abstract:
Commit messages play an important role in several software engineering tasks such as program comprehension and understanding program evolution. However, programmers neglect to write good commit messages. Hence, several Commit Message Generation (CMG) tools have been proposed. We observe that the recent state of the art CMG tools use simple and easy to compute automated evaluation metrics such as B…
▽ More
Commit messages play an important role in several software engineering tasks such as program comprehension and understanding program evolution. However, programmers neglect to write good commit messages. Hence, several Commit Message Generation (CMG) tools have been proposed. We observe that the recent state of the art CMG tools use simple and easy to compute automated evaluation metrics such as BLEU4 or its variants. The advances in the field of Machine Translation (MT) indicate several weaknesses of BLEU4 and its variants. They also propose several other metrics for evaluating Natural Language Generation (NLG) tools. In this work, we discuss the suitability of various MT metrics for the CMG task. Based on the insights from our experiments, we propose a new variant specifically for evaluating the CMG task. We re-evaluate the state of the art CMG tools on our new metric. We believe that our work fixes an important gap that exists in the understanding of evaluation metrics for CMG research.
△ Less
Submitted 20 April, 2022;
originally announced April 2022.
-
Replacements and Replaceables: Making the Case for Code Variants
Authors:
Venkatesh Vinayakarao,
Devika Sondhi,
Sumit Keswani,
Rahul Purandare,
Anita Sarma
Abstract:
There are often multiple ways to implement the same requirement in source code. Different implementation choices can result in code snippets that are similar, and have been defined in multiple ways: code clones, examples, simions and variants. Currently, there is a lack of a consistent and unambiguous definition of such types of code snippets. Here we present a characterization study of code varia…
▽ More
There are often multiple ways to implement the same requirement in source code. Different implementation choices can result in code snippets that are similar, and have been defined in multiple ways: code clones, examples, simions and variants. Currently, there is a lack of a consistent and unambiguous definition of such types of code snippets. Here we present a characterization study of code variants - a specific type of code snippets that differ from each other by at least one desired property, within a given code context. We distinguish code variants from other types of redundancies in source code, and demonstrate the significant role that they play: about 25% to 43% of developer discussions (in a set of nine open source projects) were about variants. We characterize different types of variants based on their code context and desired properties. As a demonstration of the possible use of our characterization of code variants, we show how search results can be ranked based on a desired property (e.g., speed of execution).
△ Less
Submitted 14 June, 2020; v1 submitted 6 June, 2020;
originally announced June 2020.
