Spatial-Temporal Preserving Multimodal Algorithm for EEG-based Emotion Recognition
DOI:
https://doi.org/10.54097/hset.v61i.10293Keywords:
Emotion Recognition, Human-computer Interaction, Machine Learning.Abstract
Emotion recognition has been the focus of some human computer interaction researchers. Designing an efficient algorithm for emotion recognition is crucial for its potential use in HCI. Many algorithms have been developed for emotion recognition and a majority of them is based on electroencephalography (EEG). Despite the success of spatial–temporal recurrent neural network (STRNN) and multimodal deep neural networks, they can still be improved. This paper proposes an algorithm which tries to combine STRNN with multimodal deep neural network, where EEG signals are classified using STRNN and eye movement signals are classified using SVM. The results from the two classifiers are fused by max fusion. This paper shows that by taking spatial and temporal information of EEG signals into consideration and using multimodal fusion, the performance of emotion recognition can be improved.
Downloads
References
Mauss, I. B., and Robinson, M.D., “Measures of emotion: A review, Cognition and Emotion”, 23 (2), 209 - 237 (2009).
Zheng, W. -L., Liu, W., Lu, Y., Lu, B. -L., and Cichocki, A., “EmotionMeter: A Multimodal Framework for Recognizing Human Emotions”, IEEE Transactions on Cybernetics, 49 (3), 1110 - 1122 (2019).
Egger, M., Ley, M., Hanke, S., “Emotion Recognition from Physiological Signal Analysis: A Review”, Electronic Notes in Theoretical Computer Science, 343, 35 - 55 (2019).
Ekman, P., “Universals and cultural differences in facial expressions of emotion”, Nebraska Symposium on Motivation, 19, 207 - 283 (1971).
Liu, W. et al., “Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition”, IEEE Transactions on Cognitive and Developmental Systems, 14 (2), 715-729 (2022).
Zhang, D. et al., “Making Sense of Spatio-Temporal Preserving Representations for EEG-Based Human Intention Recognition”, IEEE Transactions on Cybernetics, 50(7), 3033-3044 (2020).
Mehmood, R. M., Lee, H. J., “A novel feature extraction method based on late positive potential for emotion recognition in human brain signal patterns”, Computers and Electrical Engineering, 53, 444 - 457 (2016).
Zhang, T. et al., “Spatial–Temporal Recurrent Neural Network for Emotion Recognition”, IEEE Transactions on Cybernetics, 49 (3), 839 - 847 (2019).
Poria, S. et al., “A review of affective computing: from unimodal analysis to multimodal fusion”, Information Fusion, 37, 98 - 125 (2017).
Nakisa, B. et al., “Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors”, Expert Systems with Applications, 93, 143 - 155 (2018).
Downloads
Published
Issue
Section
License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.







