Performance Analysis of Model-Contrastive Algorithm

Authors

  • Yuantao Liao

DOI:

https://doi.org/10.54097/hset.v39i.6600

Keywords:

Federated Learning; Image Classification; Deep Neural Network; Contrastive Learning; Machine Learning.

Abstract

Federated learning develops multiple clients with a universal framework platform to train deep learning models with excellent performance. During this process, the local data set is not leaked to the outside, making federated learning a powerful method to protect data security while solving the problem of isolated data islands. However, in some usage scenarios, different clients are holding local data sets which are non- Independent Identically Distribution, i.e., non-IID. There have been significant attempts made to overcome this difficulty, and one of these efforts is the model-contrastive federated learning framework, as known as MOON algorithm. The main concept is to make use of the similarities between the global and local models to adjust the local training. The final global model performs well with non-IID data. The paper aims to replicate and analyze this algorithm to provide data reference for subsequent use. The algorithm is tested on the MNIST data set to get its performance on the handwritten digit classification task. This paper processed the data set to make it non-IID, but the result achieved a high accuracy rate with an acceptable computational cost.

Downloads

Download data is not yet available.

References

Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast-learning algorithm for deep belief nets. Neural computation, 18(7), 1527-1554.

Wang, J. Z., Kong, L. W., Huang, Z., et al. (2020). Research review of federated learning algorithms. Big Data Reserch, 6(6), 70-88.

Kairouz, P., McMahan, H. B., Avent, B., et al. (2021). Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2), 1-210.

McMahan, B., Moore, E., Ramage, D., et al. (2017, April). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282).

Karimireddy, S. P., Kale, S., Mohri, M., et al. (2019). SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning.

Li, Q., He, B., & Song, D. (2021). Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10713-10722).

Li, T., Sahu, A. K., Zaheer, M., et al. (2020). Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2, 429-450.

He, K., Fan, H., Wu, Y., et al. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9729-9738).

Chen, T., Kornblith, S., Norouzi, M., et al. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607).

Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6), 141-142.

Downloads

Published

01-04-2023