Skip to main content
@shmVirus

publications

Rubric Drift in Large Programming Courses: Measuring and Mitigating Inconsistency Across Grading Teams

Sabbir Hosen Mamun, Helena R. Novak, Daniel A. Fernández

Preprint2026Under review at ACM TOCE
PreprintDatasetCite

Abstract

Large programming courses rely on teams of teaching assistants to apply shared rubrics, but grading expectations can drift as edge cases accumulate. We introduce a rubric drift metric based on disagreement trajectories over time and evaluate it on 18,742 graded submissions from three offerings of a second-year software engineering course. Calibration checkpoints reduced late-semester disagreement by 19%, while exemplar-based norming was most effective for design-quality criteria that lacked executable tests.

Cite this work

@misc{Mamun2026Rubric,
  title = {Rubric Drift in Large Programming Courses: Measuring and Mitigating Inconsistency Across Grading Teams},
  author = {Sabbir Hosen Mamun and Helena R. Novak and Daniel A. Fernández},
  year = {2026},
  howpublished = {\url{https://arxiv.org/abs/2601.09231}},
  note = {Preprint},
  abstract = {Large programming courses rely on teams of teaching assistants to apply shared rubrics, but grading expectations can drift as edge cases accumulate. We introduce a rubric drift metric based on disagreement trajectories over time and evaluate it on 18,742 graded submissions from three offerings of a second-year software engineering course. Calibration checkpoints reduced late-semester disagreement by 19%, while exemplar-based norming was most effective for design-quality criteria that lacked executable tests.},
}