EDIT: Evidence-Diagnosed Intervention Training

for Rule-Faithful LLM Grading

Zhihao Wu*,1, Linhai Zhang*,1, Taiyi Wang*,2, Runcong Zhao1, Peter Andrews3, Cesare Aloisi3, Yulan He†,1,4
1King's College London   2University of Cambridge   3AQA   4The Alan Turing Institute
Findings of the Association for Computational Linguistics: EMNLP 2026
*Indicates Equal Contribution    †Corresponding Author
Code arXiv
A 4-mark grading question where the standard grader rejects two correct phrases due to wording mismatches, shifting its belief away from the gold mark; EDIT revises only that step and restores the correct mark of 4/4.
A single misaligned judgement derails the grade. On this 4-mark question, the grader correctly credits two criteria (S1–2), but then rejects two correct phrases due to wording mismatches (S3), shifting its belief away from the gold mark and concluding 2/4. Edit locally revises only that step (S3′), linking each disputed phrase to its rubric criterion, which pulls the belief trajectory back to the gold mark and restores the correct mark of 4/4.

Abstract

Reliable rubric grading requires more than accurate score prediction. Each judgement must be grounded in the mark scheme and evidence from the student answer. Existing credit-assignment and intervention methods, primarily designed for self-contained reasoning tasks such as mathematics reasoning, struggle in this setting because they do not identify where grading reasoning goes wrong or how the model's belief about the final mark changes during reasoning. We propose Evidence-Diagnosed Intervention Training (EDIT), a two-phase framework for training more rubric-faithful LLM graders. First, EDIT-SFT locates problematic reasoning steps using internal model signals: posterior belief over the final mark and input-grounding scores. It then revises only these local steps with help from a rubric checklist. Second, EDIT-RL calibrates the grader with belief-guided reward shaping, penalising large harmful belief drifts while still allowing helpful exploration. Experiments on two real-world, multi-subject grading benchmarks demonstrate that EDIT consistently outperforms strong supervised fine-tuning and reinforcement learning baselines on both in-domain and out-of-domain splits, with ablation studies confirming that internal-state diagnostics drive these gains. Under deterministic rubric-edit interventions, EDIT-SFT is the most rule-responsive of all evaluated systems, and EDIT-RL largely retains this responsiveness while improving accuracy.

How EDIT Works

Overview of the EDIT pipeline: internal-state signals (posterior belief probe and grounding audits) locate flawed substeps; a rubric checklist guides atomic local revisions that are regenerated, filtered, and used for SFT; the SFT policy is then calibrated with GRPO using a belief-guided shaped reward.
The EDIT pipeline. Both phases read the same internal quantity: the model's posterior belief over the final mark. Phase 1 (EDIT-SFT): for incorrect rollouts, internal-state signals — a posterior belief probe and masked-support grounding audits — locate and rank flawed substeps. A per-response rubric checklist is provided to the reviser as privileged input to produce atomic corrective edits under a locality constraint, and the corrected trajectories are aggregated for supervised fine-tuning. Phase 2 (EDIT-RL): the SFT policy is further calibrated with GRPO, augmented with a threshold-respecting, belief-guided reward computed from a frozen posterior probe that penalises harmful mid-trajectory belief excursions while tolerating benign exploration.

Results

Dataset (split) Base GRPO DGPO InT EDIT-SFT EDIT-RL
SAS — avg. of 3 subjects (OOD) 0.7270.6890.7210.731 0.7810.794
Private-Biology (ID) 0.7010.7270.7040.648 0.7380.754
Private-Biology (OOD) 0.5240.5350.5330.508 0.5480.575
Private-Physics (ID) 0.4770.4800.5350.551 0.5270.555
Private-Physics (OOD) 0.4560.4650.4490.409 0.4720.487

Grading accuracy (QWK, higher is better). Quadratic weighted kappa on the SAS benchmark (History, Geography, Physics; all questions unseen in training) and the proprietary Private-Science benchmark, with Qwen3-8B as the policy model for every method. The InT column reports its final RL stage. EDIT-RL is best on every dataset and split, and question-level cluster-bootstrap tests show its SAS gains remain significant after Holm correction while the baselines' do not.

Training dynamics in QWK on ID and OOD test sets for four combinations of base choices (Base, EDIT-SFT) and RL choices (GRPO, EDIT-RL), showing EDIT-SFT plus EDIT-RL is strongest, especially out-of-distribution.
Both phases matter. Training dynamics (QWK) of RL choices (GRPO, EDIT-RL) applied to either the base model or the EDIT-SFT checkpoint on Private-Biology. Starting from EDIT-SFT dominates starting from the base model, and belief-guided EDIT-RL gives a further consistent lift over plain GRPO — most visibly on the out-of-distribution split, where EDIT-SFT + EDIT-RL is the clear strongest combination.
Histogram of the relative positions of candidate steps selected with and without EDIT's internal-state driven localiser: the base model prefers early steps, while the localiser selects middle and final steps.
Where do graders actually go wrong? Relative position of the step selected for revision with and without EDIT's internal-state localiser. Prompt-based self-audits gravitate to early steps, whereas the localiser selects middle or final steps, where credit synthesis typically occurs — and ablations confirm this localisation is the largest single contributor to EDIT's gains.

Rule-faithfulness under interventions. Beyond accuracy, we test whether a grader truly follows the rubric by applying deterministic edits to the mark scheme whose gold-score change is known exactly, and measuring how closely the model's score change tracks it. EDIT-SFT is the most rule-responsive of all evaluated systems (mean absolute deviation from perfect responsiveness of 0.285 vs. 0.412 for the base grader), and EDIT-RL largely retains this responsiveness while improving accuracy. Points-total rescaling remains unsolved for every evaluated system, exposing absolute-count anchoring as an open challenge that accuracy metrics alone cannot detect.

BibTeX

@inproceedings{wu2026edit,
  title     = {{EDIT}: Evidence-Diagnosed Intervention Training for Rule-Faithful {LLM} Grading},
  author    = {Wu, Zhihao and Zhang, Linhai and Wang, Taiyi and Zhao, Runcong and Andrews, Peter and Aloisi, Cesare and He, Yulan},
  booktitle = {Findings of the Association for Computational Linguistics: {EMNLP} 2026},
  publisher = {Association for Computational Linguistics},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.06350}
}