Risk-Set Misalignment Inflates Apparent Performance in OBE-Oriented Landmark Temporal Prediction
DOI:
https://doi.org/10.54097/m3389688Keywords:
Outcome-based Education, Landmark Temporal Prediction, Risk-set Alignment, Trustworthy Evaluation, Machine LearningAbstract
Temporal event prediction models are widely deployed in outcome-based education (OBE) and other domains where timely intervention depends on reliable prospective estimates. However, the trustworthiness of reported performance hinges on whether the evaluation cohort is aligned with the prospective decision context-specifically, whether subjects whose outcomes have already been realized are excluded at the prediction landmark. This paper presents a systematic evaluation framework that formalizes landmark temporal prediction as a risk-set alignment task, in which eligibility, feature availability and future outcomes are rigorously anchored to a common time point. Using 32,593 enrolment records and over ten million activity logs from the Open University Learning Analytics Dataset (OULAD), we evaluate logistic regression and histogram gradient boosting at five weekly landmarks under both a prospective landmark-aligned risk set and a retrospective diagnostic cohort. The results reveal a striking discrepancy: aligned ROC-AUC ranges from 0.634 to 0.679, whereas the retrospective cohort yields 0.872 at week 12-a difference driven by simultaneous changes in population, prevalence and outcome composition. Event-time stratification confirms that already-resolved events are substantially easier to classify than future events (ROC-AUC 0.905–0.966 vs. 0.623–0.683). These findings demonstrate that risk-set misalignment inflates apparent discrimination and compromises the trustworthiness of temporal prediction models in OBE and analogous prospective decision settings. We provide a reproducible reporting protocol to support comparable evaluation across studies.
Downloads
References
[1] Kuzilek, J., Hlosta, M., & Zdrahal, Z. (2017). Open university learning analytics dataset. Scientific Data, 4, 170171. https://doi.org/10.1038/sdata.2017.171 DOI: https://doi.org/10.1038/sdata.2017.171
[2] He, Y., Chen, R., Li, X., Hao, C., Liu, S., Zhang, G., & Jiang, B. (2020). Online at risk student identification using RNN GRU joint neural networks. Information, 11(10), 474. https://doi.org/10.3390/info11100474 DOI: https://doi.org/10.3390/info11100474
[3] Liu, Y., Fan, S., Xu, S., Sajjanhar, A., Yeom, S., & Wei, Y. (2023). Predicting student performance using clickstream data and machine learning. Education Sciences, 13(1), 17. https://doi.org/10.3390/educsci13010017 DOI: https://doi.org/10.3390/educsci13010017
[4] Jawad, K., Shah, M. A., & Tahir, M. (2022). Students’ academic performance and engagement prediction in a virtual learning environment using random forest with data balancing. Sustainability, 14(22), 14795. https://doi.org/10.3390/su142214795 DOI: https://doi.org/10.3390/su142214795
[5] Zhang, C., & Ahn, H. (2023). E learning at risk group prediction considering the semester and realistic factors. Education Sciences, 13(11), 1130. https://doi.org/10.3390/educsci13111130 DOI: https://doi.org/10.3390/educsci13111130
[6] Shou, Z., Xie, M., Mo, J., & Zhang, H. (2024). Predicting student performance in online learning: A multidimensional time series data analysis approach. Applied Sciences, 14(6), 2522. https://doi.org/10.3390/app14062522 DOI: https://doi.org/10.3390/app14062522
[7] Ujkani, B., Minkovska, D., & Hinov, N. (2024). Course success prediction and early identification of at risk students using explainable artificial intelligence. Electronics, 13(21), 4157. https://doi.org/10.3390/electronics13214157 DOI: https://doi.org/10.3390/electronics13214157
[8] Qiu, F., Zhang, G., Sheng, X., Jiang, L., Zhu, L., Xiang, Q., Jiang, B., & Chen, P. K. (2022). Predicting students’ performance in e learning using learning process and behaviour data. Scientific Reports, 12(1), 453. https://doi.org/10.1038/s41598 021 03867 8 DOI: https://doi.org/10.1038/s41598-021-03867-8
[9] Bilal, M., Omar, M., Anwar, W., Bokhari, R. H., & Choi, G. S. (2022). The role of demographic and academic features in a student performance prediction. Scientific Reports, 12(1), 12508. https://doi.org/10.1038/s41598 022 15880 6 DOI: https://doi.org/10.1038/s41598-022-15880-6
[10] Poudyal, S., Mohammadi Aragh, M. J., & Ball, J. E. (2022). Prediction of student academic performance using a hybrid 2D CNN model. Electronics, 11(7), 1005. https://doi.org/10.3390/electronics11071005 DOI: https://doi.org/10.3390/electronics11071005
[11] Qiu, F., Zhu, L., Zhang, G., Sheng, X., Ye, M., Xiang, Q., & Chen, P. K. (2022). E learning performance prediction: Mining the feature space of effective learning behavior. Entropy, 24(5), 722. https://doi.org/10.3390/e24050722 DOI: https://doi.org/10.3390/e24050722
[12] Bettahi, A., Belouadha, F. Z., & Harroud, H. (2025). A modular and explainable machine learning pipeline for student dropout prediction in higher education. Algorithms, 18(10), 662. https://doi.org/10.3390/a18100662 DOI: https://doi.org/10.3390/a18100662
Downloads
Published
Issue
Section
License
Copyright (c) 2026 International Journal of Education and Humanities
This work is licensed under a Creative Commons Attribution 4.0 International License.
