A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups
DOI:
https://doi.org/10.54097/g70yrm22Keywords:
Automated essay scoring, AI-assisted assessment, teacher-centered evaluation, rubric consistency, formative feedback, algorithmic fairness, learning to defer, conformal prediction.Abstract
AI-assisted assessment is increasingly used to score student writing and to generate formative feedback, yet it is still validated mainly by aggregate agreement with consensus scores. For classroom adoption the decisive question is teacher-centered: does the AI apply the rubric the way a competent teacher does, diagnose the same weaknesses, and treat every student group alike? We propose a Teacher-Centered Evaluation (TCE) framework with four lenses—teacher-relative rubric consistency, rubric-guided counterfactual tests, feedback diagnostic validity, and score-conditional fairness—complemented by a teacher-in-the-loop routing policy with group-audited conformal intervals. TCE is instantiated on two public corpora released by the research team behind the ASAP 2.0 benchmark: PERSUADE 2.0 (5,531 argumentative essays with holistic scores, 55,068 expert-rated discourse elements, and student demographics) and ELLIPSE (6,462 essays double-scored by 27 trained raters on seven rubric traits). Five AI scoring engines are audited. The strongest engine reaches a quadratic weighted kappa (QWK) of 0.870 and, in a teacher-substitution test, agrees with a teacher almost exactly as well as a second teacher does (ΔQWK = −0.016). TCE nevertheless exposes failures that agreement statistics hide: scores are blind to sentence order, rise by 0.27 points when a paragraph is duplicated, and fall no more when evidence is deleted than when equally long non-evidence text is removed; analytic trait scores collapse into a single dimension (inter-trait r = 0.87 vs. 0.61 for teachers); feedback misses 70% of the weaknesses that experts flag in English-learner writing; and conditioning on the teacher’s score uncovers subgroup gaps that raw statistics mask. Routing 20% of essays to teachers with a learned deferral model raises QWK to 0.907, reduces the worst subgroup bias by 31%, and nearly halves the English-learner agreement gap.
Downloads
References
[1] Crossley, S. A., Baffour, P., Burleigh, L., & King, J. (2025). A large-scale corpus for assessing source-based writing quality: ASAP 2.0. Assessing Writing, 65, 100954. DOI: https://doi.org/10.1016/j.asw.2025.100954
[2] Crossley, S. A., et al. (2024). A large-scale corpus for assessing written argumentation: PERSUADE 2.0. Assessing Writing, 61, 100865. DOI: https://doi.org/10.1016/j.asw.2024.100865
[3] Crossley, S. A., Baffour, P., Tian, Y., Picou, A., Benner, M., & Boser, U. (2022). The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (PERSUADE) corpus 1.0. Assessing Writing, 54, 100667. DOI: https://doi.org/10.1016/j.asw.2022.100667
[4] Crossley, S. A., et al. (2023). Measuring second language proficiency using the English Language Learner Insight, Proficiency and Skills Evaluation (ELLIPSE) corpus. International Journal of Learner Corpus Research, 9(2), 248–269. DOI: https://doi.org/10.1075/ijlcr.22026.cro
[5] Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. DOI: https://doi.org/10.1111/j.1745-3992.2011.00223.x
[6] Shermis, M. D. (2014). State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration. Assessing Writing, 20, 53–76. DOI: https://doi.org/10.1016/j.asw.2013.04.001
[7] Perelman, L. (2014). When “the state of the art” is counting words. Assessing Writing, 21. DOI: https://doi.org/10.1016/j.asw.2014.05.001
[8] Bridgeman, B., Trapani, C., & Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1), 27–40. DOI: https://doi.org/10.1080/08957347.2012.635502
[9] Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3).
[10] Taghipour, K., & Ng, H. T. (2016). A neural approach to automated essay scoring. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1882–1891). DOI: https://doi.org/10.18653/v1/D16-1193
[11] Ke, Z., & Ng, V. (2019). Automated essay scoring: A survey of the state of the art. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI) (pp. 6300–6308). DOI: https://doi.org/10.24963/ijcai.2019/879
[12] Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. DOI: https://doi.org/10.1016/j.rmal.2023.100050
[13] Pack, A., Barrett, A., & Escalante, J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6, 100234. DOI: https://doi.org/10.1016/j.caeai.2024.100234
[14] Tate, T. P., et al. (2024). Can AI provide useful holistic essay scoring? Computers and Education: Artificial Intelligence, 7, 100255. DOI: https://doi.org/10.1016/j.caeai.2024.100255
[15] Xiao, C., Ma, W., Xu, S. X., Zhang, K., Wang, Y., & Fu, Q. (2024). Human-AI collaborative essay scoring: A dual-process framework with LLMs. arXiv:2401.06431. DOI: https://doi.org/10.1145/3706468.3706507
[16] Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. DOI: https://doi.org/10.3102/003465430298487
[17] Steiss, J., et al. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. DOI: https://doi.org/10.1016/j.learninstruc.2024.101894
[18] Dai, W., et al. (2023). Can large language models provide feedback to students? A case study on ChatGPT. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT) (pp. 323–325). DOI: https://doi.org/10.1109/ICALT58122.2023.00100
[19] Loukina, A., Madnani, N., & Zechner, K. (2019). The many dimensions of algorithmic fairness in educational applications. In Proceedings of the 14th Workshop on Innovative Use of NLP for Building Educational Applications (BEA) (pp. 1–10). DOI: https://doi.org/10.18653/v1/W19-4401
[20] Loukina, A., et al. (2020). Using PRMSE to evaluate automated scoring systems in the presence of label noise. In Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications (BEA). DOI: https://doi.org/10.18653/v1/2020.bea-1.2
[21] Baker, R. S., & Hawn, A. (2022). Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052–1092. DOI: https://doi.org/10.1007/s40593-021-00285-9
[22] Romano, Y., Barber, R. F., Sabatti, C., & Candès, E. (2020). With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2(2). DOI: https://doi.org/10.1162/99608f92.03f00592
[23] Madras, D., Pitassi, T., & Zemel, R. (2018). Predict responsibly: Improving fairness and accuracy by learning to defer. In Advances in Neural Information Processing Systems 31 (NeurIPS).
[24] Mozannar, H., & Sontag, D. (2020). Consistent estimators for learning to defer to an expert. In Proceedings of the 37th International Conference on Machine Learning (ICML) (pp. 7076–7087). PMLR 119.
[25] Geifman, Y., & El-Yaniv, R. (2017). Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30 (NeurIPS).
[26] Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 4902–4912). DOI: https://doi.org/10.18653/v1/2020.acl-main.442
[27] Ding, Y., Riordan, B., Horbach, A., Cahill, A., & Zesch, T. (2020). Don’t take “nswvtnvakgxpm” for an answer—The surprising vulnerability of automatic content scoring systems to adversarial input. In Proceedings of the 28th International Conference on Computational Linguistics (COLING) (pp. 882–892). DOI: https://doi.org/10.18653/v1/2020.coling-main.76
[28] Madnani, N., & Loukina, A. (2016). RSMTool: Collection of tools building and evaluating automated scoring models. Journal of Open Source Software, 1(3), 33. DOI: https://doi.org/10.21105/joss.00033
[29] Ke, G., et al. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems 30 (NIPS) (pp. 3146–3154).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Academic Journal of Management and Social Sciences

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

