How Consistent Are Humans When Grading Programming Assignments?
Authors:
Marcus Messer,
Neil C. C. Brown,
Michael Kölling,
Miaojing Shi
Abstract:
Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the class leader, larger cohorts are often assessed by multiple assessors, which increases the risk of inconsistent grading. To investigate the consistency…
▽ More
Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the class leader, larger cohorts are often assessed by multiple assessors, which increases the risk of inconsistent grading. To investigate the consistency of human grading of programming assignments, we asked 28 participants to each grade 40 CS1 introductory Java assignments, providing grades and feedback for correctness, code elegance, readability and documentation; the 40 assignments were split into two batches of 20. In the second batch of 20, we duplicated one assignment from the first to analyse the internal consistency of individual assessors. We measured the inter-rater reliability of the groups using Krippendorf's $α$ -- an $α> 0.667$ is recommended to make tentative conclusions based on the rating. Our groups were inconsistent, with an average $α= 0.2$ when grading correctness and an average $α< 0.1$ for code elegance, readability and documentation. To measure the individual consistency of graders, we measured the distance between the grades they awarded for the duplicated assignment in batch one and batch two. Only one participant of the 22 who didn't notice that the assignment was a duplicate was awarded the same grade for correctness, code elegance, readability and documentation. The average grade difference was 1.79 for correctness and less than 1.6 for code elegance, readability and documentation. Our results show that human graders in our study can not agree on the grade to give a piece of student work and are often individually inconsistent, suggesting that the idea of a ``gold standard'' of human grading might be flawed, and highlights that a shared rubric alone is not enough to ensure consistency.
△ Less
Submitted 28 February, 2025; v1 submitted 2 September, 2024;
originally announced September 2024.
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Authors:
Marcus Messer,
Neil C. C. Brown,
Michael Kölling,
Miaojing Shi
Abstract:
We conducted a systematic literature review on automated grading and feedback tools for programming education.
We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation and evaluation techniques.
Most papers assess the correctness of assignments in object-oriented languages.
Typically, these to…
▽ More
We conducted a systematic literature review on automated grading and feedback tools for programming education.
We analysed 121 research papers from 2017 to 2021 inclusive and categorised them based on skills assessed, approach, language paradigm, degree of automation and evaluation techniques.
Most papers assess the correctness of assignments in object-oriented languages.
Typically, these tools use a dynamic technique, primarily unit testing, to provide grades and feedback to the students or static analysis techniques to compare a submission with a reference solution or with a set of correct student submissions.
However, these techniques' feedback is often limited to whether the unit tests have passed or failed, the expected and actual output, or how they differ from the reference solution.
Furthermore, few tools assess the maintainability, readability or documentation of the source code, with most using static analysis techniques, such as code quality metrics, in conjunction with grading correctness.
Additionally, we found that most tools offered fully automated assessment to allow for near-instantaneous feedback and multiple resubmissions, which can increase student satisfaction and provide them with more opportunities to succeed.
In terms of techniques used to evaluate the tools' performance, most papers primarily use student surveys or compare the automatic assessment tools to grades or feedback provided by human graders.
However, because the evaluation dataset is frequently unavailable, it is more difficult to reproduce results and compare tools to a collection of common assignments.
△ Less
Submitted 5 December, 2023; v1 submitted 20 June, 2023;
originally announced June 2023.
