Stability of a Large Language Model-Based Automated Grading System: A Diagnostic Analysis of a Multi-Phase Architecture with Constrained Revision

Abstract

Large language models (LLMs) can interpret open-ended responses and generate scores with written rationales. Their outputs may vary across calls, even when the assignment, rubric, model, and sampling settings remain unchanged. This variation can arise from prompt design, context organization, sampling, or the service environment. It may become more difficult to trace in a grading workflow that includes initial scoring, score revision, and holistic review, because a later phase can either moderate or enlarge an earlier difference. This study investigates that problem through a three-phase grading workflow: ensemble-based initial scoring, self-critique-based revision, and holistic review. System logs were examined to identify where score differences emerged. The logs suggest that incompatible structured-output formats may interfere with the aggregation of initial scores. They also show that the revision and review phases can make relatively large changes when no explicit adjustment limit is imposed. In response, the system combined prompt-level guidance with programmatic clipping. In Phases 2 and 3, the adjustment for each dimension was restricted to ±30% of the corresponding score produced in the preceding phase. The diagnostic sample consisted of one student assignment graded by DeepSeek-Chat. Six baseline tests and four tests after the constraint mechanism was added were conducted. Among the two baseline tests with complete phase-level records, the final scores differed by 5.80 points. Most of this difference appeared during self-critique-based revision and holistic review. The four post-deployment final scores were 5.40, 4.50, 3.70, and 4.20, producing a range of 1.70 points, or 11.3% of the assignment’s maximum score. The narrower range suggests that adjustment limits can reduce large swings in the total score in this workflow. The control was less successful at the item level: Phase 3 still showed more cross-test variation than Phase 1, and only four of the twenty scoring items received the same score in all four tests. These results are consistent with prior discussions of LLM output variability, but they also indicate that total-score stability can conceal changes in individual scoring dimensions. The evidence is limited to one assignment, one model, and ten repeated tests, so the findings should be read as a system diagnosis rather than a general estimate of grading reliability. Further tests across assignments, models, rubrics, and sampling settings are needed before the mechanism can be applied more broadly.