Mutation Testing via Iterative Large Language Model-Driven Scientific Debugging
Authors:
Philipp Straubinger,
Marvin Kreis,
Stephan Lukasczyk,
Gordon Fraser
Abstract:
Large Language Models (LLMs) can generate plausible test code. Intuitively they generate this by imitating tests seen in their training data, rather than reasoning about execution semantics. However, such reasoning is important when applying mutation testing, where individual tests need to demonstrate differences in program behavior between a program and specific artificial defects (mutants). In t…
▽ More
Large Language Models (LLMs) can generate plausible test code. Intuitively they generate this by imitating tests seen in their training data, rather than reasoning about execution semantics. However, such reasoning is important when applying mutation testing, where individual tests need to demonstrate differences in program behavior between a program and specific artificial defects (mutants). In this paper, we evaluate whether Scientific Debugging, which has been shown to help LLMs when debugging, can also help them to generate tests for mutants. In the resulting approach, LLMs form hypotheses about how to kill specific mutants, and then iteratively generate and refine tests until they succeed, all with detailed explanations for each step. We compare this method to three baselines: (1) directly asking the LLM to generate tests, (2) repeatedly querying the LLM when tests fail, and (3) search-based test generation with Pynguin. Our experiments evaluate these methods based on several factors, including mutation score, code coverage, success rate, and the ability to identify equivalent mutants. The results demonstrate that LLMs, although requiring higher computation cost, consistently outperform Pynguin in generating tests with better fault detection and coverage. Importantly, we observe that the iterative refinement of test cases is important for achieving high-quality test suites.
△ Less
Submitted 11 March, 2025;
originally announced March 2025.
3rd Workshop on Maritime Computer Vision (MaCVi) 2025: Challenge Results
Authors:
Benjamin Kiefer,
Lojze Žust,
Jon Muhovič,
Matej Kristan,
Janez Perš,
Matija Teršek,
Uma Mudenagudi Chaitra Desai,
Arnold Wiliem,
Marten Kreis,
Nikhil Akalwadi,
Yitong Quan,
Zhiqiang Zhong,
Zhe Zhang,
Sujie Liu,
Xuran Chen,
Yang Yang,
Matej Fabijanić,
Fausto Ferreira,
Seongju Lee,
Junseok Lee,
Kyoobin Lee,
Shanliang Yao,
Runwei Guan,
Xiaoyu Huang,
Yi Ni
, et al. (23 additional authors not shown)
Abstract:
The 3rd Workshop on Maritime Computer Vision (MaCVi) 2025 addresses maritime computer vision for Unmanned Surface Vehicles (USV) and underwater. This report offers a comprehensive overview of the findings from the challenges. We provide both statistical and qualitative analyses, evaluating trends from over 700 submissions. All datasets, evaluation code, and the leaderboard are available to the pub…
▽ More
The 3rd Workshop on Maritime Computer Vision (MaCVi) 2025 addresses maritime computer vision for Unmanned Surface Vehicles (USV) and underwater. This report offers a comprehensive overview of the findings from the challenges. We provide both statistical and qualitative analyses, evaluating trends from over 700 submissions. All datasets, evaluation code, and the leaderboard are available to the public at https://macvi.org/workshop/macvi25.
△ Less
Submitted 17 January, 2025;
originally announced January 2025.
