An Overview of Data Contribution Evaluation Methods for Federated Learning

Chenxi Ding; Mingchuan Zhang

doi:10.54097/mhwz3z95

Authors

Chenxi Ding
Mingchuan Zhang

DOI:

https://doi.org/10.54097/mhwz3z95

Keywords:

Data Evaluation; Fair Incentive; Federated Learning; Machine Learning.

Abstract

With the rapid development of artificial intelligence, especially machine learning technology, the demand for data is also increasing. As a distributed learning method, federated learning (FL) allows total multiple parties to co-train models locally while protecting data privacy. Due to heterogeneity in data, devices, and other factors, the contribution of participants to model quality also varies. If these differences are ignored, it poses a challenge to achieving a fair FL environment. Therefore, fairly assessing the contribution levels of each data provider becomes a critical issue in FL. This plays a significant role in fairly motivating participants and promoting the sustainable development of FL. This paper reviews existing evaluation methods from the perspective of data contribution assessment in FL, discusses ongoing challenges in this area, and explores future research directions.

Downloads

Download data is not yet available.

References

Huynh-The T, Pham Q V, Pham X Q, et al. Artificial intelligence for the metaverse: A survey[J]. Engineering Applications of Artificial Intelligence, 2023, vol. 117(105581).

Nilsson N J. The quest for artificial intelligence[M]. Cambridge, UK, Cambridge University Press, 2009.

Dilmaghani S, Brust M R, Danoy G, et al. Privacy and security of big data in AI systems: A research and standards perspective[C]. 2019 IEEE international conference on big data. Los Angeles, CA, USA, 2019, pp. 5737-5743.

Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology, 2019, vol. 10(2), pp. 1-19.

McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C] In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Fort Lauderdale, USA, 2017, vol. 54, pp. 1273–1282.

Kairouz P, McMahan H B, Avent B, et al. Advances and open problems in federated learning[J]. Foundations and trends® in machine learning, 2021, vol. 14(1–2), pp. 1-210.

Duan Q, Huang J, Hu S, et al. Combining federated learning and edge computing toward ubiquitous intelligence in 6G network: Challenges, recent advances, and future directions[J]. IEEE Communications Surveys & Tutorials, 2023.

Rahman A, Hossain M S, Muhammad G, et al. Federated learning-based AI approaches in smart healthcare: concepts, taxonomies, challenges and open issues[J]. Cluster computing, 2023, vol. 26(4), pp. 2271-2311.

Nguyen D C, Pham Q V, Pathirana P N, et al. Federated learning for smart healthcare: A survey[J]. ACM Computing Surveys (Csur), 2022, vol. 55(3), pp. 1-37.

T. Li, M. Sanjabi, and V. Smith, Fair resource allocation in federated learning[C]. In Proceedings of the 8th International Conference on Learning Representations. Addis Ababa, Ethiopia, 2020.

Ezzeldin Y H, Yan S, He C, et al. Fairfed: Enabling group fairness in federated learning[C]. In Proceedings of the 34th AAAI Conference on Artificial Intelligence. Washington, DC, USA, 2023, vol. 37(6), pp. 494-7502.

Sim R H L, Xu X, Low B K H. Data Valuation in Machine Learning:" Ingredients", Strategies, and Open Challenges[C]. In Proceedings of the 31st International Joint Conference on Artificial Intelligence. Vienna, Austria, 2022, pp. 5607-5614.

Zhang T, Mao S. An introduction to the federated learning standard [J]. GetMobile: Mobile Computing and Communications, 2022, vol. 25(3), pp.18-22.

Bottou L. Large-scale machine learning with stochastic gradient descent[C]. In Proceedings of the 19th International Conference on Computational Statistics. Paris, France, 2010, pp. 177-186.

Chen Y, Yang X, Qin X, et al. Dealing with label quality disparity in federated learning[J]. Federated Learning: Privacy and Incentive, 2020, pp. 108-121.

Banica T, Curran S. Decomposition results for Gram matrix determinants[J]. Journal of mathematical physics, 2010, vol. 51(11), p. 113503.

Xu X, Wu Z, Foo C S, et al. Validation free and replication robust volume-based data valuation[C]. In Proceedings of 35th International Conference on Neural Information Processing Systems. 2021, vol. 34, pp. 10837–10848.

Jiang M, Roth H R, Li W, et al. Fair federated medical image segmentation via client contribution estimation[C]. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 16302-16311.

Wang Y, Li G L, and Li K Y. Survey on contribution evaluation for federated learning[J]. Ruan Jian Xue Bao, 2022, vol. 34(3), pp.1168–1192.

Kang J, Xiong Z, Niyato D, et al. Incentive mechanism for reliable federated learning: A joint optimization approach to combining reputation and contract theory[J]. IEEE Internet of Things Journal, 2019, vol. 6(6), pp.10700-10714.

Miller N, Resnick P, Zeckhauser R. Eliciting informative feedback: The peer-prediction method[J]. Management Science, 2005, vol. 51(9), pp. 1359-1373.

Lv H, Zheng Z, Luo T, et al. Data-free evaluation of user contributions in federated learning[C] In Proceedings of 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks. 2021, pp. 81-88.

Cook R D, Weisberg S. Characterizations of an empirical influence function for detecting influential cases in regression[J]. Technometrics, 1980, vol. 22(4), pp. 495-508.

Richardson A, Filos-Ratsikas A, Faltings B. Rewarding high-quality data via influence functions[J]. arXiv preprint arXiv: 1908.11598, 2019.

Shyn S K, Kim D, Kim K. Fedccea: A practical approach of client contribution evaluation for federated learning[J]. arXiv preprint arXiv: 2106.02310, 2021.

Xu X, Lyu L, Ma X, et al. Gradient driven rewards to guarantee fairness in collaborative machine learning[J]. In Proceedings of 35th International Conference on Neural Information Processing Systems. 2021, pp. 16104–16117.

Cawley G C, Talbot N L C. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers[J]. Pattern recognition, 2003, vol. 36(11), pp. 2585-2592.

Koh P W, Liang P. Understanding black-box predictions via influence functions[C]. In Proceedings of the 34th International Conference on Machine Learning. Sydney, NSW, Australia, 2017, vol. 70, pp. 1885–189.

Shapley L S. A value for n-person games. In: Contributions to the Theory of Games [M]. Princeton, New Jersey, Princeton University Press, 1953, vol. 2(28), pp. 307–318.

Sim R H L, Zhang Y, Chan M C, et al. Collaborative machine learning with incentive-aware model rewards[C]. In Proceedings of the 37th International Conference on Machine Learning. 2020, pp. 8927–8936.

Fan Z, Fang H, Zhou Z, et al. Improving fairness for data valuation in horizontal federated learning[C]. In Proceedings of the 38th International Conference on Data Engineering. Kuala Lumpur, Malaysia, 2022, pp. 2440-2453.

Fan Z, Fang H, Zhou Z, et al. Fair and efficient contribution valuation for vertical federated learning[J]. arXiv preprint arXiv: 2201.02658, 2022.

Jia R, Dao D, Wang B, et al. Towards efficient data valuation based on the shapley value[C]. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics. Naha, Okinawa, Japan, 2019, pp. 1167-1176.

Wang T, Rausch J, Zhang C, et al. A principled approach to data valuation for federated learning[J]. Federated Learning: Privacy and Incentive, 2020, pp. 153-167.

Kwon Y, Rivas M A, Zou J. Efficient computation and analysis of distributional shapley values[C]. In Proceedings of 24th the International Conference on Artificial Intelligence and Statistics. 2021, vol. 130, pp. 793-801.

Yan T, Procaccia A D. If you like shapley then you’ll love the core[C]. In Proceedings of the 33th AAAI Conference on Artificial Intelligence. 2021, vol. 35(6), pp. 5751–5759.

Villani C. Optimal transport: old and new[M]. Berlin: springer, 2009, vol. 338.

Peyré G, Cuturi M. Computational optimal transport: With applications to data science[J]. Foundations and Trends® in Machine Learning, 2019, vol. 11(5-6), pp. 355-607.

Just H A, Kang F, Wang J T, et al. Lava: Data valuation without pre-specified learning algorithms[C]. In Proceedings of the 7th International Conference on Learning Representations. Kigali, Rwanda, 2023.

Rakotomamonjy A, Nadjahi K, Ralaivola L. Federated Wasserstein Distance [J]. arXiv preprint arXiv: 2310.01973, 2023.