Research on Federated Learning Algorithms Driven by Data Heterogeneity

Xiaoyi Zhao; Xinchen Zhang; Shasha Wang

doi:10.54097/0htgq955

Authors

Xiaoyi Zhao
Xinchen Zhang
Shasha Wang

DOI:

https://doi.org/10.54097/0htgq955

Keywords:

Federated Learning, Data Heterogeneity, Three-dimensional Optimization Framework, Adaptive Optimization

Abstract

Federated Learning as a distributed machine learning paradigm enables collaborative modeling among multiple participants while preserving data privacy. However, challenges such as model convergence difficulties and low communication efficiency caused by client-side data heterogeneity remain critical bottlenecks hindering its practical applications. This paper constructs a three-dimensional analytical framework encompassing "client-local optimization, server aggregation strategies, and global convergence guarantees" based on mathematical characterization of data heterogeneity. Through systematic analysis of core research achievements, we reveal evolutionary patterns of key technical approaches including dynamic learning rate adaptation, gradient correction mechanisms, and heterogeneity-aware regularization. The study further identifies three fundamental challenges: multi-objective optimization dilemmas, inadequate adaptability to dynamic data drift, and theory-practice gaps. Future breakthroughs should focus on cross-modal knowledge transfer architectures and trusted federated learning mechanisms to enable reliable algorithm deployment in open environments.

Downloads

Download data is not yet available.

References

[1] Yang Q, Liu Y, Cheng Y, et al. Federated Learning [M]. Beijing: Electronics Industry Press, 2020: 2-5.

[2] Peng N, Wang H. Federated Learning Technology and Practice [M]. Beijing: Electronics Industry Press, 2021: 340.

[3] Konecny J, McMahan H B, Yu F X, et al. Federated Learning: Strategies for Improving Communication Efficiency[J]. arXiv preprint arXiv: 1610.05492, 2016.

[4] Li T, Sahu A K, Talwalkar A, et al. Federated Learning: Challenges, Methods, and Future Directions[J]. IEEE Signal Processing Magazine, 2020, 37(3): 50-56

[5] Zhao, Y., Li, M., Lai, L., et al. 2018. Federated Learning with Non-IID Data[J]. arXiv preprint arXiv:1806.00582.

[6] McMahan H B, Moore E, Ramage D, et al. Communication-Efficient Learning of Deep Networks from Decentralized[C] in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017, pp. 1273-1282.

[7] Li X, Huang KX, Yang WS, et al. On the Convergence of FedAvg on Non-IID Data[C] In the 8th International Conference on Learning Representations (ICLR), 2020, pp. 1-26.

[8] Karimireddy S P, Kale S, Mohri M, et al. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning[C]In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020, pp. 5123-5143.

[9] Yu H, Fan J, Sun Y, et al. A Survey on Statistical Heterogeneity in Federated Learning [J/OL]. Computer Applications,1-11[2025-03-06].http://kns.cnki. net/kcms/detail/ 51.1307.TP. 2024 1226.1339.002.html.

[10] Li X, Huang K, Yang W, et al. On the Convergence of FedAvg on Non-IID Data[C] In the 8th International Conference on Learning Representations (ICLR), 2020, pp. 1-26.

[11] Pan Z, Wang Z, Li C, et al. Federated Unlearning with Gradient Descent and Conflict Mitigation[C] in the Proceedings of the 39th AAAI Conference on Artificial Intelligence. arXiv preprint arXiv: 2412.20200, 2024.

[12] Li Y. Simulation Study of Heterogeneity in High Toughness Polymer Network Structures [D]. University of Science and Technology of China,2024. DOI:10.27517/d. cnki. gzkju. 2024. 000093.

[13] Smith V, Chiang C K, Sanjabi M, et al. Federated Multi-Task Learning[C] in the 30th Conference on Neural Information Processing Systems (NeurIPS), 2017, pp. 4424-4434.

[14] Li D, Wang J. FedMD: Heterogenous Federated Learning via Model Distillation[J]. arXiv preprint arXiv: 1910.03581, 2019.

[15] Chen F, Luo M, Dong Z, et al. Federated Meta-Learning with Fast Convergence and Efficient Communication[J]. arXiv preprint arXiv:1802.07876, 2018.

[16] Li T, Sahu A K, Zaheer M, et al. Federated Optimization in Heterogeneous Networks[C] in Proceedings of Machine Learning and Systems (MLSys), 2020.

[17] Wang J, Liu Q, Liang H, et al. A Novel Framework for the Analysis and Design of Heterogeneous Federated Learning[J]. IEEE Transactions on Signal Processing, 2021, 69: 5234-5249.

[18] Zhu Z, Hong J, Zhou J. Data-Free Knowledge Distillation for Heterogeneous Federated learning[C] International Conference on Machine Learning, 2021, pp. 12878-12889.

[19] Wiedemann S, Muller K R, Samek W. Compact and Computationally Efficient Representation of Deep Neural Networks [J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 31(3): 772-785.

[20] Zhao Y, Chen JJ, Guo QL, et al. Network Anomaly Detection Using Federated Learning and Transfer Learning[C] in the First International Conference on Security and Privacy in Digital Economy (SPDE), 2020, pp. 219-231.

[21] Reddi S, Charles Z, Zaheer M, et al. Adaptive Federated Optimization[J]. arXiv preprint arXiv: 2003.00295, 2020.

[22] Fallah, A., Mokhtari, A., Ozdaglar, A. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach [C]In Advances in Neural Information Processing Systems 33 (NeurIPS), 3557–3568.

[23] Li, Q., He, B., Song, D. Model-Contrastive Federated Learning [C]In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 10713–10722. DOI:10.1109/CVPR46437.2021.01057

[24] Khanduri, P., Sharma, P., Kafle, S. Distributed Stochastic Non-Convex Optimization: Momentum-Based Variance Reduction [J]. arXiv preprint arXiv: 2005.00224, 2020.

[25] Wang, J., Liu, Q., Liang, H., et al. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization[C]In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020, 7611–7623.

[26] Acar, D.A., Zhao, Y., Matas, R., et al. Federated Learning Based on Dynamic Regularization[C]In Proceedings of the 9th International Conference on Learning Representations (ICLR), 2021.

[27] Lin, T., Kong, L., Stich, S.U., et al. Ensemble Distillation for Robust Model Fusion in Federated Learning[C]In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022, 3151–3163.

[28] Koloskova, A., Loizou, N., Boreiri, S., et al. A Unified Theory of Decentralized SGD with Changing Topology and Local Updates [J]. Journal of Machine Learning Research (JMLR), 2022, 23(1):1–50.

[29] Fu, J., Chen Z, Han X. 2022. Adap DP-FL: Differentially Private Federated Learning with Adaptive Noise[C]. IEEE International Conference on Trust, 2022, 656--663.