Optimization and Application of Distributed Matrix Factorization Algorithm in Recommendation Systems

Huang Yan

doi:10.54097/4t6drh76

Authors

Huang Yan School of civil engineering, Shenyang Jianzhu university, Shenyang, China

DOI:

https://doi.org/10.54097/4t6drh76

Keywords:

Distributed, Matrix Decomposition, Collaborative Filtering, Attribute Fusion, Large-scale Data Processing.

Abstract

The development of the Internet and big data technologies has led to an exponential increase in the scale of users, items, and interaction data in recommendation systems. Traditional single-machine matrix factorization algorithms suffer from low computational efficiency, memory bottlenecks, and poor scalability. This paper studies the optimization and implementation of distributed matrix factorization algorithms in recommendation systems. Using the MovieLens-32M dataset as the experimental object, it analyzes the performance shortcomings of traditional centralized algorithms, builds a distributed matrix factorization framework based on stochastic gradient descent, integrates data parallel partitioning, L2 regularization constraints, item attribute feature fusion, and distributed training optimization strategies, and solves the problems of long training time, insufficient feature expression, and overfitting in large-scale scenarios. Distributed model comparison experiments are conducted using RMSE, MAE, and training time as indicators. The results show that the optimized algorithm not only ensures the accuracy of rating prediction but also significantly improves the training efficiency of large-scale data, with a significant reduction in RMSE compared to the basic distributed model and a significant decrease in training time, verifying its practicality and superiority in large-scale distributed recommendation systems.

References

[1] Xiang L. Recommendation System Practice. Posts & Telecom Press, Beijing, 2012.

[2] Xu H L, Wu X, Li X D, et al. Comparative Research on Internet Recommender Systems. Journal of Software, 2009, 20(2): 350-362.

[3] Deng A L, Zhu Y Y, Shi B L. Collaborative Filtering Recommendation Algorithm Based on Item Rating Prediction. Journal of Software, 2003, 14(9): 1621-1628.

[4] Zhou A Y, Jin C Q, Wang G R. Distributed Data Management and Parallel Processing Technology. Science Press, Beijing, 2021.

[5] Takács G, Pilászy I, Németh B, et al. Matrix Factorization and Neighbor Based Algorithms the Netflix Prize Problem. In: Proceedings of the 2008 ACM Conference on Recommender Systems. New York: ACM, 2008: 267-274.

[6] Wang K Y, Wang Z Q, Shi J X. Review of the Frontier Progress of Recommender Systems and Their Application in the Field of Design. Design, 2025, 38(21): 135-139.

[7] Yan Y J. Optimization of Personalized Marketing Recommendation Algorithm Based on Big Data. Market Modernization, 2025(18): 27-29.

[8] Zhu W B. Research and Implementation of Improved Collaborative Filtering Recommendation Algorithm. Nanjing University of Posts and Telecommunications, Nanjing, 2024.

[9] Koren Y, Bell R, Volinsky C. Matrix Factorization Techniques for Recommender Systems. Computer, 2009, 42(8): 30-37.

[10] Li H. Statistical Learning Methods. Tsinghua University Press, Beijing, 2019.

[11] Oliveira L D. Distributed Bayesian Personalized Ranking in Spark. Stanford: Stanford University, 2016.

[12] Jesus B, Jorge D, Fernando O, et al. Comprehensive Evaluation of Matrix Factorization Models for Collaborative Filtering Recommender Systems. arXiv:2410.17644. 2024.