Impact of Comments on LLM Comprehension of Legacy Code
Authors:
Rock Sabetto,
Emily Escamilla,
Devesh Agarwal,
Sujay Kandwal,
Justin F. Brunelle,
Scott Rosen,
Nitin Naik,
Samruddhi Thaker,
Eric O. Scott,
Jacob Zimmer,
Amit Madan,
Arun Sridharan,
Doug Wendt,
Michael Doyle,
Christopher Glasz,
Jasper Phillips,
William Macke,
Colin Diggs,
Michael Bartholf,
Zachary Robin,
Paul Ursino
Abstract:
Large language models (LLMs) have been increasingly integrated into software engineering and maintenance tasks due to their high performance with software engineering tasks and robust understanding of modern programming languages. However, the ability of LLMs to comprehend code written with legacy languages remains a research gap challenged by real-world legacy systems lacking or containing inaccu…
▽ More
Large language models (LLMs) have been increasingly integrated into software engineering and maintenance tasks due to their high performance with software engineering tasks and robust understanding of modern programming languages. However, the ability of LLMs to comprehend code written with legacy languages remains a research gap challenged by real-world legacy systems lacking or containing inaccurate documentation that may impact LLM comprehension. To assess LLM comprehension of legacy languages, there is a need for objective LLM evaluation. In order to objectively measure LLM comprehension of legacy languages, we need an efficient, quantitative evaluation method. We leverage multiple-choice question answering (MCQA), an emerging LLM evaluation methodology, to evaluate LLM comprehension of legacy code and the impact of comment prevalence and inaccurate comments. In this work, we present preliminary findings on the impact of documentation on LLM comprehension of legacy code and outline strategic objectives for future work.
△ Less
Submitted 23 April, 2025;
originally announced June 2025.
Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation
Authors:
Colin Diggs,
Michael Doyle,
Amit Madan,
Siggy Scott,
Emily Escamilla,
Jacob Zimmer,
Naveed Nekoo,
Paul Ursino,
Michael Bartholf,
Zachary Robin,
Anand Patel,
Chris Glasz,
William Macke,
Paul Kirk,
Jasper Phillips,
Arun Sridharan,
Doug Wendt,
Scott Rosen,
Nitin Naik,
Justin F. Brunelle,
Samruddhi Thaker
Abstract:
Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electron…
▽ More
Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems.
△ Less
Submitted 22 November, 2024;
originally announced November 2024.
