Parallel EM algorithm on Hadoop for large-scale hidden Markov model parameter estimation
DOI:
https://doi.org/10.54097/hset.v77i.14396Keywords:
Hadoop, Parallel EM algorithm, Hidden Markov model, Parameter estimation.Abstract
This paper aims to propose a parallel EM algorithm based on Hadoop for parameter estimation of large-scale hidden Markov models (HMM). HMM is a commonly used statistical model. However, since the parameter estimation of HMM involves the storage and processing of large-scale data sets, traditional serial algorithms have certain limitations in efficiency. This paper introduces the Hadoop parallel computing framework, divides the task into multiple subtasks through the MapReduce programming model and assigns them to different machines for parallel computing, which improves the efficiency and scalability of parameter estimation. The results show that using parallel EM algorithm for large-scale hidden Markov model parameter estimation on Hadoop is feasible and effective.
Downloads
References
Gillick D, Faria A, DeNero J. Mapreduce: Distributed computing for machine learning[J]. Berkley, Dec, 2006, 18.
Cao Xu. Research and Improvement of a Massive Log Data Processing Model on the Hadoop Platform [D] Zhejiang University of Technology, 2013.
Lu Jiang, Li Yun Research on Feature Selection Parallelization Based on MapReduce [J] Computer Science, 2015, 42 (08): 44-47.
Li Zhao, Li Xiao, Wang Chunmei, et al Research on a MapReduce based text clustering method [J] Computer Science, 2016, 43 (01): 246-250.
He Qing, Li Ning, Luo Wenjuan, et al Overview of Machine Learning Algorithms under Big Data [C]//Conference on Artificial Intelligence of the Chinese Computer Society two thousand and thirteen
Lin J, Dyer C. Data-intensive text processing with MapReduce[M]. Springer Nature, 2022.
Shim, Kyuseok. "MapReduce algorithms for big data analysis." International Workshop on Databases in Networked Information Systems. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013.
Karloff H, Suri S, Vassilvitskii S. A model of computation for mapreduce[C]//Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms. Society for Industrial and Applied Mathematics, 2010: 938-948.
Wolfe J A, Haghighi A, Klein D. Fully distributed EM for very large datasets[C]//ICML. 2008, 8: 1184-1191.
.Su Jiageng. Distributed EM Clustering Algorithm Based on Hadoop Platform [D] Hebei Normal University, 2014.
Gupta M R, Chen Y. Theory and use of the EM algorithm[J]. Foundations and Trends® in Signal Processing, 2011, 4(3): 223-296.
Wang Aiping, Zhang Gongying, Liu Fang Research and Application of EM Algorithm [J] Computer Technology and Development, 2009, 19 (9): 108-110.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
