Advances and Challenges in Multi-Modal Emotion Recognition: A Comprehensive Investigation

Abstract

Multi-Modal Emotion Recognition (MER) combines information from two or more modalities—such as speech, facial expressions, video, and physiological signals—to more accurately infer people’s emotional states. Previous work shows that relying on a single modality often misses important cues: for instance, audio may capture tone but not facial micro-expressions, video may capture expression but not internal arousal. Recent systems using CNNs, Transformers, or hybrid fusion architectures, applied in driving-safety and healthcare contexts, have improved accuracy significantly, especially when handling missing or noisy modalities. This survey reviews such methods, discusses current challenges like interpretability, modality mismatch, and real-time deployment, and suggests future directions including lightweight models, privacy-preserving fusion, and cross-domain generalization. Furthermore, it highlights the growing importance of explainable and adaptive models that can dynamically adjust to changing environments and user contexts. As MER continues to evolve, these innovations will enhance emotion-aware applications in human-computer interaction, mental health, and intelligent systems.