Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Jiaqi Sun

doi:10.54097/agvjvq19

Authors

Jiaqi Sun

DOI:

https://doi.org/10.54097/agvjvq19

Keywords:

Multimodal Emotion Recognition, Human-Computer Interaction, Psychological Analysis, Machine Learning Methods.

Abstract

In this study, the focus is primarily on the diverse methods for recognizing human emotions through language, text, and facial expressions via computational technology. Emphasizing the real-world applicability of these techniques, the paper underscores the significance of multimodal emotion recognition in areas such as human-computer interaction, psychology, and emotion analytics. Multimodal methods, which combine data from various sources like voice tone, facial cues, and textual context, offer a robust approach for discerning nuanced emotional states. Compared to single-mode analysis, these multimodal techniques tend to produce more accurate and comprehensive results, bridging the gaps left by any one mode in isolation. As technology increasingly integrates with daily human activity, the importance of nuanced, reliable emotion recognition is becoming paramount for fostering more natural and empathic machine-human interactions. Moreover, in the realm of psychology, these methods offer groundbreaking possibilities for diagnosis and treatment. By discussing the future applications and methodologies of multimodal emotion recognition, this paper aims to provide a comprehensive roadmap for both academic research and practical applications in the evolving landscape of emotion-aware computing.

Downloads

Download data is not yet available.

References

Abdullah, S. M. S. A., Ameen, S. Y. A., Sadeeq, M. A., & Zeebaree, S. (2021). Multimodal emotion recognition using deep learning. Journal of Applied Science and Technology Trends, 2(02), 52-58. DOI: https://doi.org/10.38094/jastt20291

Chen, J., Wang, C., Wang, K., Yin, C., Zhao, C., Xu, T., ... & Yang, T. (2021). HEU Emotion: a large-scale database for multimodal emotion recognition in the wild. Neural Computing and Applications, 33, 8669-8685. DOI: https://doi.org/10.1007/s00521-020-05616-w

Tan, Y., Sun, Z., Duan, F., Solé-Casals, J., & Caiafa, C. F. (2021). A multimodal emotion recognition method based on facial expressions and electroencephalography. Biomedical Signal Processing and Control, 70, 103029. DOI: https://doi.org/10.1016/j.bspc.2021.103029

Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020, April). M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 02, pp. 1359-1367). DOI: https://doi.org/10.1609/aaai.v34i02.5492

Chuang, Z. J., & Wu, C. H. (2004, August). Multi-modal emotion recognition from speech and text. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 9, Number 2, August 2004: Special Issue on New Trends of Speech and Language Processing (pp. 45-62).

Cai, L., Dong, J., & Wei, M. (2020, November). Multi-modal emotion recognition from speech and facial expression based on deep learning. In 2020 Chinese automation congress (CAC) (pp. 5726-5729). IEEE. DOI: https://doi.org/10.1109/CAC51589.2020.9327178

Xu, H., Zhang, H., Han, K., Wang, Y., Peng, Y., & Li, X. (2019). Learning alignment for multimodal emotion recognition from speech. arXiv preprint arXiv:1909.05645. DOI: https://doi.org/10.21437/Interspeech.2019-3247

Siriwardhana, S., Kaluarachchi, T., Billinghurst, M., & Nanayakkara, S. (2020). Multimodal emotion recognition with transformer-based self-supervised feature fusion. IEEE Access, 8, 176274-176285. DOI: https://doi.org/10.1109/ACCESS.2020.3026823

Tripathi, S., Tripathi, S., & Beigi, H. (2018). Multi-modal emotion recognition on iemocap dataset using deep learning. arXiv preprint arXiv:1804.05788.

Sharma, G., & Dhall, A. (2021). A survey on automatic multimodal emotion recognition in the wild. Advances in data science: Methodologies and applications, 35-64. DOI: https://doi.org/10.1007/978-3-030-51870-7_3