Multimodal Humanoid Robotic Interaction System
DOI:
https://doi.org/10.54097/1a5vmt30Keywords:
Multimodal interaction; humanoid robot; vision-language model; human-robot collaboration; embodied intelligence.Abstract
The interactive system of multimodal humanoid robots is a pivotal technology for achieving robotic intelligence. This paper presents a systematic review of the key cutting-edge technologies, difficulties, challenges, and development prospects pertaining to such interactive systems. Firstly, the paper enumerates the core technologies in this field, including the application of visual-language models in robot training, long-context multimodal command navigation, learning to imitate human movements, force and torque sensing control, and multimodal intent fusion technology. Insights into the current challenges faced by the field are also derived from these technologies, such as complex system integration, weak environmental adaptability, insufficient safety, and limitations in hardware performance. Finally, the paper looks ahead to the future development of multimodal humanoid robots, proposing that the in-depth integration of large models with embodied intelligence, multimodal interaction, and the coordinated development of software and hardware are the keys to realizing robotic intelligence. The paper argues that the interactive system of multimodal humanoid robots will significantly enhance the level of robotic intelligence, find wide application in various fields, and greatly drive the development of productivity.
References
[1]Liu P, Orru Y, Vakil J, Paxton C, Shafiullah NM, Pinto L. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401.12202. 2024 Jan 22.
[2]Chiang HT, Xu Z, Fu Z, Jacob MG, Zhang T, Lee TW, Yu W, Schenck C, Rendleman D, Shah D, Xia F. Mobility vla: Multimodal instruction navigation with long-context vlms and topological graphs. arXiv preprint arXiv:2407.07775. 2024 Jul 10.
[3]Fu Z, Zhao Q, Wu Q, Wetzstein G, Finn C. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406.10454. 2024 Jun 15.
[4]Zhou Q, Feng B, Li B, Liu C, Chen Y, Bi Y. Enhancing Human-Robot Collaborative Transportation of Deformable Objects using Multi-modal Reinforcement Learning and Adaptive Admittance Control. Advanced Engineering Informatics. 2026 Jan 1;69:103905.
[5]Jiang S, Zhang W, Liu J, Li H, Wang Z, Zhou Y, He B. A novel human-in-the-loop multimodal intention fusion method for human-robot interaction. IEEE Transactions on Automation Science and Engineering. 2025 Jun 17.
[6]Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, Du Y. A survey of large language models. arXiv preprint arXiv:2303.18223. 2023 Mar 31;1(2).
[7]Chen J, Chen Z. Extended Bayesian information criteria for model selection with large model spaces. Biometrika. 2008 Sep 1;95(3):759-71.
[8]Cangelosi A, Bongard J, Fischer MH, Nolfi S. Embodied intelligence. InSpringer handbook of computational intelligence 2015 (pp. 697-714). Berlin, Heidelberg: Springer Berlin Heidelberg.
[9]Gupta A, Savarese S, Ganguli S, Fei-Fei L. Embodied intelligence via learning and evolution. Nature communications. 2021 Oct 6;12(1):5721.
[10]Stiefelhagen R, Ekenel HK, Fugen C, Gieselmann P, Holzapfel H, Kraft F, Nickel K, Voit M, Waibel A. Enabling multimodal human–robot interaction for the karlsruhe humanoid robot. IEEE Transactions on Robotics. 2007 Oct 8;23(5):840-51.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
