Stacking Multiple Machine Learning Ways and Using Optuna to Precisely Predict Students' GPA

Chuyi Qu

doi:10.54097/08aaz166

Authors

Chuyi Qu

DOI:

https://doi.org/10.54097/08aaz166

Keywords:

GPA; Stacking; Optuna: Catboost; LightGBM.

Abstract

The application of the Internet in the field of education and teaching is increasingly widespread, and there are massive educational data generated in this process. How to make reasonable use of these massive educational data has always been an important issue in the field of educational data mining. A student's Grade Point Average (GPA) is crucial for evaluating their own development, helping teachers plan teaching, and enabling schools to formulate education programs. Although there have been many precedents of using machine learning to predict students' GPA, the fitting process is often relatively simple. The method adopted in this study, which comprehensively utilizes Stacking and Optuna. Tune the hyperparameters of the base models using Optuna to enhance their fitting capabilities, successfully leverages the advantages of both, the coefficient of determination (R²) reaches 0.88. And this way is more accurate than previous model construction methods, demonstrating the potential and prospects of this method in the field of regression fitting.

References

[1]A. Joshi, P. Saggar, R. J. Moolchand, S. Deepak, G. A. Khanna. “CatBoost - An Ensemble Machine Learning Model for Prediction and Classification of Student Academic Performance.” Advances in Data Science and Adaptive Analysis: Theory and Applications, 13(4):2141002-1-2141002-28 (2021).

[2]L. Prokhorenkova, G. Gusev, A. Vorobev, et al. “CatBoost: unbiased boosting with categorical features.” Advances in neural information processing systems, 31. (2018).

[3]B. Albreiki, N. Zaki, H. Alashwal. “A Systematic Literature Review of Student’ Performance Prediction Using Machine Learning Techniques.” Educ. Sci. 11, 552. (2021).

[4]R. Arifuddin, et al. "Effectiveness of Machine Learning Models with Bayesian Optimization-Based Method to Identify Important Variables that Affect GPA." Jurnal Teknologi dan Aplikasi Matematika (JTAM), vol. 8, no. 3, art. 21711. (2023).

[5]T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. “Optuna: A Next-generation Hyperparameter Optimization Framework.” In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19). Association for Computing Machinery, New York, NY, USA, 2623–2631. (2019).

[6]J. M. Ahn, J. Kim, K. Kim. “Ensemble Machine Learning of Gradient Boosting (XGBoost, LightGBM, CatBoost) and Attention-Based CNN-LSTM for Harmful Algal Blooms Forecasting.” Toxins. 15, 608. (2023).

[7]G. Ke, Q. Meng, T. Finley, et al. “Lightgbm: A highly efficient gradient boosting decision tree.” Advances in neural information processing systems, 30. (2017).

[8]T. O. Hodson. “Root means square error (RMSE) or mean absolute error (MAE): When to use them or not.” Geoscientific Model Development Discussions, 1-10. (2022).

[9]N. J. D. Nagelkerke. “A note on a general definition of the coefficient of determination.” biometrika, 78(3): 691-692. (1991).

[10]J. R. Yu, X. M. Chang, S. H. Hu, H. D. Yin, J. J. Wu. “Combining travel behavior in metro passenger flow prediction: A smart explainable Stacking-Catboost algorithm,” Information Processing & Management, Volume 61, Issue 4, 103733, ISSN 0306-4573, (2024).

[11]S. Džeroski, B. Ženko. “Is combining classifiers with stacking better than selecting the best one?” Machine learning, 54(3): 255-273. (2004).