Research on the Construction of an Automation Platform Based on Data Stream Processing

Lele Li

doi:10.54097/r8mfam56

Authors

Lele Li

DOI:

https://doi.org/10.54097/r8mfam56

Keywords:

Data Stream Processing, Automation Platform, Machine Learning, Zero-mean, Flink

Abstract

With the deepening of digital transformation, to address the problems of high latency and low accuracy in processing industrial production automation data using traditional methods, this paper proposes a type of research on the construction of an automation platform based on data stream processing. The research is analyzed in a four-layer structure. Firstly, a machine learning method is adopted to perform zero-mean standardization on the accessed data; secondly, the stream processing engine layer is constructed using Apache Flink technology to process data streams continuously with low latency; then, the storage service layer of the platform is built based on metadata; finally, it is explained that the data operation, maintenance and application layer is responsible for the stable operation of the system and data output. The research results provide theoretical support and practical reference for enterprises to build an efficient, reliable, and easy-to-use real-time data automation platform.

Downloads

Download data is not yet available.

References

[1] Zhang Shouli, Liu Chen. Research on Service-oriented Cloud-edge-end Collaborative Data Stream Processing Architecture [J]. Journal of Shandong Agricultural University (Natural Science Edition), 2024, 55(03): 385-395.

[2] Chen Juan. Research on Real-time Data Stream Processing and Analysis Strategy Based on Big Data [C]// China Technology Market Association, China High-Tech Industrialization Research Association, China International Association for Science and Technology Cooperation, Entrepreneurs Branch of China Future Research Association, Discover Magazine. Proceedings of the 23rd China Scientists Forum. Shanghai Cairongju Information Technology Co., Ltd.; 2024: 91-98. DOI: 10. 26914/c.cnkihy.2024.053511.

[3] Xi Rongkang, Cai Manchun, Lu Tianliang. Tor Traffic Analysis Model Based on Data Augmentation and Data Stream Processing [J]. Computer Engineering, 2023, 49(03): 177-184. DOI: 10.19678/j.issn.1000-3428.0064386.

[4] Y.Jing, W. Yafei and Z. Fan, Research and Application Strategy for Intelligent Car Platform Construction Based on Flink, Kafka Stream Data Technology and Deepseek, 2025 IEEE 3rd International Conference on Image Processing and Computer Applications (ICIPCA), Shenyang, China, 2025, pp. 1-5, doi: 10.1109/ICIPCA65645.2025.11139039.

[5] X.Wang, J. Lu, F. Zhang and J. Yang, Automobile Brand Analysis System Based on Feature Engineering and Apache Kafka+Flink Stream Data Processing Framework, 2025 International Conference on Computer Science, Technology and Engineering (ICCSTE), Wuhan, China, 2025, pp. 128-133, doi: 10.1109/ICCSTE65902.2025.11138357.

[6] Wang, Y., Zhang, F., Feng, Q. et al. Strategic analysis of intelligent connected vehicle industry competitiveness: a comprehensive evaluation system integrating rough set theory and projection pursuit. Complex Intell.Syst. 10, 7033–7062 (2024). https://doi.org/10.1007/s40747-024-01525-w.

[7] Zhao Xiaoyu, Yang Xing, Bu Lei. Research on SQLIA Recognition Technology Combining Machine Learning and Feature Engineering [J]. Computer Programming Skills & Maintenance, 2025, (08): 129-132. DOI:10.16184/j. cnki. comprg. 2025.08.030.

[8] Kai, G. Yaxin and Z. Fan, Research on the Application of Flink Streaming Data Technology in the Construction of Automobile Internationalization Platform, 2025 Asia-Europe Conference on Cybersecurity, Internet of Things and Soft Computing (CITSC), Rimini, Italy, 2025, pp. 778-782, doi: 10.1109/ CIT SC64390.2025.00145.

[9] Y. Liu, Y. Wang and X. Yang, Research on the Construction of a Distributed Overseas Data Acquisition and Preprocessing Platform Based on the FLINK Real-Time Streaming Computing Framework, 2025 International Conference on Computer Science, Technology and Engineering (ICCSTE), Wuhan, China, 2025, pp. 01-07, doi: 10.1109/ ICCSTE 65902. 2025. 11138049.