Key Technologies and Applications of Multimodal Cognitive Agent Construction in the Vertical Domain of Inspection and Testing
DOI:
https://doi.org/10.54097/jx4e5b29Keywords:
Multimodal Agent, Large Language Model, Inspection and Testing, Cognitive Computing, Knowledge GraphAbstract
Inspection and testing underpin product safety and regulatory compliance across industries such as manufacturing, healthcare and food and beverage. However, conventional engineering test processes that mainly depend on manual, offline actions, failed to adapt to next generation, automated, digital and environmental-sensitive testing, resulting in very low productivity and high cost for most firms. The automation and digitalization of inspection and testing processes have become a research hotspot in both academia and industry. The natural language processing and computer interpretation of test results have been a focus of AI research as well. Nonetheless, due to the lack of real data, the verification and simulation of real inspection and testing environments are still difficult for researchers. As a solution, we manage to develop a multimodal cognitive test system that fuses the textual regulatory documents and instrument data, through a series of modules and processing flows. Our test system is based on a so-called multimodal cognitive agent, which includes large language model, vision module, knowledge graph and retrieval-augmented generation. We introduce the design, development and application of our test system, which used for rubber heater in a glass factory, and layout the future challenges for the exploration of multimodal agent technology in the test engineering. The project is supported by the Guangxi Key Research and Development Program. This paper was partially presented at the IEEE 2023 International Conference on Intelligent Commerce (ICIC).
Downloads
References
[1] Sun, H., Wang, T., Yuan, L., et al. (2026). From structure to synergy: A survey of vision-language perception paradigm evolution in multimodal large language models. Information Fusion, 133, 104285. https://doi.org/ 10.1016/j. inffus. 2026. 104285.
[2] López, V. S., Olivas, S. E., & Montañés, S. M. (2026). RGB ensemble strategies for unsupervised industrial anomaly detection on the AutoVI dataset. Electronics, 15(10), 2077. https://doi.org/10.3390/electronics15102077.
[3] Qu, J., Wang, Y., Fu, Y., et al. (2026). A multi-agent AI framework for explainable battery system maintenance. Cell Reports Physical Science, 7(6), 103388. https://doi. org/10. 1016/ j. xcrp.2026.103388.
[4] Geetika, D., Vanita, A., & Ravibabu, M. (2023). Artificial neural network based sub-surface defect detection in glass fiber reinforced polymers: Nondestructive evaluation 4.0. Sensing and Imaging, 24(1). https://doi.org/10.1007/s12217-023-00987-9.
[5] Liu, S., Sun, Y., Zhang, L., et al. (2021). Fault diagnosis of shipboard medium-voltage DC power system based on machine learning. International Journal of Electrical Power and Energy Systems, 124. https://doi.org/10.1016/ j.ijepes. 2020. 106345.
[6] Yao, L., Fang, Z., Xiao, Y., et al. (2021). An intelligent fault diagnosis method for lithium battery systems based on grid search support vector machine. Energy, 214, 118866. https:// doi. org/10.1016/j.energy.2020.118866.
[7] Meng, A., Zhang, W., & Qi, M. (2026). A semi-supervised multi-stage pipeline for low-resource industrial condition monitoring by injecting domain knowledge via cross-modal multimodal large language model fine-tuning. Engineering Applications of Artificial Intelligence, 181, 115250. https:// doi. org/ 10.1016/j.engappai.2026.115250.
[8] Qi, W., Yongsheng, H., & Feng, C. (2021). Deepening the IDA* algorithm for knowledge graph reasoning through neural network architecture. Neurocomputing, 429, 101–109. https:// doi. org/10.1016/j.neucom.2021.01.024.
[9] Baptista, L. M., Henriques, M. E., & Goebel, K. (2021). More effective prognostics with elbow point detection and deep learning. Mechanical Systems and Signal Processing, 146. https:// doi.org/10.1016/j.ymssp.2020.106989.
[10] Mao, J., Liu, J., & Shi, T. (2026). Task-guided dynamic attention mechanism for industrial anomaly detection with large vision-language models. Concurrency and Computation: Practice and Experience, 38(10), e70774. https://doi.org/ 10. 1002/ cpe.70774.
[11] Zhang, X., Zhang, L., Wang, P., et al. (2026). KPIs anomaly detection through dual-branch spatio-temporal-frequency feature extraction and enhanced transformer. International Journal of Machine Learning and Cybernetics, 17(7), 341. https: // doi.org/10.1007/s13048-026-02189-6.
[12] Haha, B. A. M., Bohli, A., Haddour, N., et al. (2026). A systematic review of industrial IoT anomaly detection and the forensic interpretability gap. Electronics, 15(11), 2240. https:// doi. org/10.3390/electronics15112240.
[13] Zhang, C., Zhang, Q. M., Luo, Y., et al. (2026). Hard constraints and soft learning dual-graph anomaly detection for industrial processes. Engineering Applications of Artificial Intelligence, 177, 114927. https://doi.org/10. 1016/j. engappai. 2026. 114927.
[14] Xu, R. H., Zhang, N., Yin, Y. Z., et al. (2025). Multimodal framework integrating multiple large language model agents for intelligent geotechnical design. Automation in Construction, 176, 106257. https://doi.org/10.1016/j.autcon.2025.106257.
[15] Guo, Q., Jiang, L., Dong, K., et al. (2026). KG-Anchored RAG: Retrieval-augmented generation for power system professional documents integrating topic modeling and knowledge graphs. Electronics, 15(11), 2362. https://doi.org/ 10.3390/ electronics 15112362.
[16] Songping, L., Feifei, L., Yusen, Y., et al. (2020). Nondestructive evaluation 4.0: Ultrasonic intelligent nondestructive testing and evaluation for composites. Research in Nondestructive Evaluation, 31(5–6), 370–388. https://doi. org/ 10.1080/0934984X.2020.1784964.
[17] Alhazmi, H. M. (2026). An explainable AI framework (XAI-CoffeeNet) with deep learning and vision transformers for Saudi coffee disease recognition and severity assessment. Smart Agricultural Technology, 14, 102259. https://doi.org/ 10. 1016/j.smartat.2026.102259.
[18] Brunaccioni, L., Fusco, D. F., & Vallerio, M. (2026). Dynamic SHAP: Time resolved explainable AI for batch processes. Computers and Chemical Engineering, 213, 109717. https:// doi. org/10.1016/j.compchemeng.2026.109717.
[19] Xiao, Y., Yin, K., & Mostafavi, A. (2026). CrisiSense-RAG: Crisis sensing multimodal retrieval-augmented generation for rapid disaster impact assessment. Computer-Aided Civil and Infrastructure Engineering, 48, 100096. https:// doi. org/ 10. 1111/ mice.13096.
[20] Chen, X., Li, Y., Bi, Y., et al. (2026). ReflectRAG: Enhancing retrieval-augmented generation with GRPO-optimized iterative reflection. Neurocomputing, 695, 134047. https://doi. org/ 10. 1016/j.neucom.2026.134047.
[21] Karabowicz, P., Charkiewicz, R., Charkiewicz, A., et al. (2026). Target-aware molecule SMILES generation using a large language model with retrieval-augmented generation, multi-turn memory, and a predictive model. Journal of Computer-Aided Molecular Design, 40(1), 134. https://doi.org/ 10.1007/ s10822-025-00489-9.
[22] Kasireddy, E., Chow, C., & Pourrahmat, M. M. (2025). Evaluating the application of GPT-4o and retrieval-augmented generation (RAG) for assessing risk of bias and study quality in systematic literature reviews (SLRs). Value in Health, 28(6S1), S278. https://doi.org/10.1016/j.jval.2025.04.018.
[23] Raghavan, P., Vaidhyanathan, K., & Blumen, R. (2025). Abhinav Kimothi on retrieval-augmented generation. IEEE Software, 42(6), 116–119. https://doi.org/10. 1109/MS. 2025. 3412476.
[24] Hai, W., Jinrui, L., & Jianfeng, D. (2020). Iterative visual relationship detection via commonsense knowledge graph. Big Data Research, 23, 100175. https://doi.org/10. 1016/j.bdr. 2020. 100175.
[25] Wang, Z. W., Li, L. B., & Wei, L. S. (2026). An integrated multimodal sensing and data fusion system for rapid, community-scale inspection of self-built housing safety hazards. Advanced Engineering Informatics, 76, 104864. https://doi.org/10.1016/j.aei.2026.104864.
[26] Ding, M., & Dong, S. (2026). Digentity (Digital-Entity) human dyad: A perspective on human-AI collaboration and decision-making. Australasian Marketing Journal, 34(2), 120–125. https: //doi.org/10.1016/j.ausmj.2026.03.002.
[27] Pan, X., Wang, D., & Tsung, F. (2026). Empowering intelligent quality control with large models: A comprehensive survey of methods, challenges, and perspectives. Frontiers of Engineering Management, 13(1), 42–64. https://doi. org/ 10. 1016/ j.fem.2026.01.003.
[28] Dong, L., Chen, Y., Ke, W., et al. (2026). Multimodal fake news detection via evidence retrieval and visual forensics with large vision-language models. Information, 17(4), 317. https:// doi. org/10.3390/info17040317.
[29] Rashidin, S. M., Taheri, B., & Javed, S. (2026). Frugal AI as a catalyst for smart hospitality: Rethinking digital transformation in resource-constrained settings. International Journal of Hospitality Management, 139, 104785. https://doi.org/ 10. 1016/ j.ijhm.2026.104785.
[30] Fernández, D. M., Prieto, R. C. J., & Cabrales, L. Á. (2026). Digital transformation and social sustainability: The mediating role of employees' digital competencies. Technovation, 156, 103607. https://doi.org/10.1016/j.technovation.2026.103607.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
