When Are Linear Classifiers Sufficient? A Re-evaluation and Conditional Boundary Exploration Based on OpenML
DOI:
https://doi.org/10.54097/xssbv193Keywords:
Linear Classifiers, Non-linear Classifiers, Algorithm Selection, Meta-learning, OpenML BenchmarkAbstract
The point at which a linear classifier stops being "sufficient" lacks a quantitative definition. To address this, we evaluated 1,687 records from 298 OpenML datasets across three model families. Using a practical significance framework, we defined linear sufficiency as an accuracy drop within a tolerance threshold δ. At δ=0.02, linear sufficiency was observed in 68.9% of SVM datasets, 62.4% of NN datasets, but only 46.1% of DT datasets. Shallow decision tree extraction highlighted low feature-to-instance ratios and small class counts as primary predictors. However, leave-one-out validation exposed a clear split. The extracted rules worked well for the DT family (0.686 accuracy vs. 0.539 baseline) but failed to surpass baselines for SVM and NN. This suggests that while simple meta-features can predict linear sufficiency for decision trees, they cannot fully capture the complex boundaries of SVMs and neural networks.
Downloads
References
[1] Strang, B., van der Putten, P., van Rijn, J. N., & Hutter, F. (2018). Don't rule out simple models prematurely: A large scale benchmark comparing linear and non-linear classifiers in OpenML. In Advances in intelligent data analysis XVII (pp. 303–315). Springer. DOI: https://doi.org/10.1007/978-3-030-01768-2_25
[2] Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014). OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), 49–60. DOI: https://doi.org/10.1145/2641190.2641198
[3] Post, M. J., van der Putten, P., & van Rijn, J. N. (2016). Does feature selection improve classification? A large scale experiment in OpenML. In International Symposium on Intelligent Data Analysis (pp. 158–170). Springer. DOI: https://doi.org/10.1007/978-3-319-46349-0_14
[4] Goodman, B., & Flaxman, S. (2017). European Union regulations on algorithmic decision-making and a "right to explanation". AI Magazine, 38(3), 50–57. DOI: https://doi.org/10.1609/aimag.v38i3.2741
[5] Dem?ar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.
[6] Nambo, H., Otsuka, A., Kimura, H., & Ueda, Y. (2016). Automatic classifier algorithm selection using optimized meta-features. Studies in Science and Technology, 5(2), 179–184.
[7] Rice, J. R. (1976). The algorithm selection problem. Advances in Computers, 15, 65–118. DOI: https://doi.org/10.1016/S0065-2458(08)60520-3
[8] Brazdil, P., Carrier, C. G., Soares, C., & Vilalta, R. (2009). Metalearning: Applications to data mining. Springer. DOI: https://doi.org/10.1007/978-3-540-73263-1
[9] Kopf, A., et al. (2024). The right to an explanation in practice: Insights from case law for the GDPR and the AI Act. In Proc. of the 2024 ACM Conference on Fairness, Accountability, and Transparency.
[10] Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.
[11] Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and regression trees. Wadsworth.
[12] Hutter, F., Kotthoff, L., & Vanschoren, J. (2019). Automated machine learning: Methods, systems, challenges. Springer. DOI: https://doi.org/10.1007/978-3-030-05318-5
[13] van Rijn, J. N., & Hutter, F. (2017). Hyperparameter importance across datasets. In Proc. of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 2367–2376). ACM. DOI: https://doi.org/10.1145/3219819.3220058
[14] Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F. (2015). Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems (pp. 2962–2970).
[15] Sculley, D., Snoek, J., Wiltschko, A., & Rahimi, A. (2018). Winner's curse? On pace, progress, and empirical rigor. In Proc. of ICLR 2018.
[16] Bischl, B., Casalicchio, G., Feurer, M., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., & Vanschoren, J. (2017). OpenML benchmarking suites and the OpenML100. arXiv preprint arXiv:1708.03731.
[17] Maldonado, S., Vairetti, C., & Figueroa, I. (2025). One-step learning algorithm selection for classification via convolutional neural networks. Information Sciences, 721, 122610. DOI: https://doi.org/10.1016/j.ins.2025.122610
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
