Enhancing Emotion Recognition in Video Characters Through Multi-Modal Deep Learning Approaches

Abstract

Human emotion recognition is a classic problem for Convolutional Neural Networks (CNNs) in classification tasks, and a plethora of research has emerged in recent years. This paper presents a video character emotion analysis model utilizing the The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) dataset, exploring the integration of audio-visual data for emotion recognition. CNNs were employed to analyze image and audio data separately, then the models were fused using linear summation and greedy selection methods. The multi-modal approach significantly enhances performance over single-modal models, with the linear summation method showing the best results at a 0.4 weight ratio. The model's predictions are visualized to demonstrate the composite emotional analysis of video characters. The findings indicate that multi-modal models can improve accuracy, precision, recall, and F1-score by 30-40 percent compared to single-modal models. Future work will focus on optimizing model performance and leveraging confusion matrices to further refine the greedy selection algorithm.