Image Emotion Analysis Combining Attention Mechanism and Multi-level Correlation
DOI:
https://doi.org/10.54097/fcis.v6i1.12Keywords:
Multi-level Features, Mixed Attention, Global and Local, BiLSTMAbstract
The development of social network has brought a large amount of image information, and the research on image emotion has gradually attracted wide attention. The current image emotion analysis methods based on multi-level features simply splice the features at each level and then classify the emotions, which not only ignores the correlation between features at different levels, but also ignores the synergistic effect between global features and local features. Therefore, this paper proposes an emotion model (MAML) based on mixed attention and multi-level dependence of images, which uses spatial and channel attention mechanisms to extract local emotion region features of images. Bi-directional Long Short Term Memory network (BiLSTM) is used to establish correlation between multi-level image global features. The experimental results of MAML model on artphoto and abstract data sets prove the validity of MAML model.
Downloads
References
Machajdik J, Hanbury A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM international conference on Multimedia. 2010: 83-92.
Yanulevskaya V, van Gemert J C, RothK, et al. Emotional valence categorization using holistic image features[C]//2008 15th IEEE international conference on Image Processing. IEEE, 2008:101-104.
Zhao S, Gao Y, Jiang X, et al. Exploring principles-of-art features for image emotion recognition[C]//Proceedings of the 22nd ACM international conference on Multimedia. 2014: 47-56.
Xu L, Wang Z, Wu B, et al. Mdan: Multi-level dependent attention network for visual emotion analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 9479-9488.
Rao T, Li X, Zhang H, et al. Multi-level region-based convolutional neural network for image emotion classification [J]. Neurocomputing, 2019, 333: 429-439.
Li B, Ren H, Jiang X, et al. SCEP—A new image dimensional emotion recognition model based on spatial and channel-wise attention mechanisms[J]. IEEE Access,2021, 9: 25278-25290.
Peng K C, Chen T, Sadovnik A, et al. Amixed bag of emotions: Model, predict, and transfer emotion distributions [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 860-868.
Compton R J. The interface between emotion and attention: A review of evidence from psychology and neuroscience[J]. Behavioral and cognitive neuroscience reviews, 2003, 2(2): 115-129.
You Q, Jin H, Luo J. Visual sentiment analysis by attending on local image regions[C]//Proceedings of the AAAI conference on artificial intelligence. 2017, 31(1).
Zhao S, Yao X, Yang J, et al. Affective image content analysis: Two decades review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(10): 6729-6751.
Song K, Yao T, Ling Q, et al. Boosting image sentiment analysis with visual attention[J]. Neurocomputing,2018, 312: 218-228.
Yang J, She D, Lai Y K, et al. Weakly supervised coupled networks for visual sentiment analysis[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7584-7592.
Chen L, Zhang H, Xiao J, et al. Sca-cnn:Spatial and channel-wise attention in convolutional networks for image captioning [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 5659-5667.
Zhu X, Li L, Zhang W, et al. Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition [C]// IJCAI. 2017: 3595-3601.
Siersdorfer S, Minack E, Deng F, et al. Analyzing and predicting sentiment of images on the social web[C]// Proceedings of the 18th ACM international conference on Multimedia. 2010: 71 5-718.
Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009: 248-255.
You Q, Luo J, Jin H, et al. Building a large scale dataset for image emotion recognition:The fine print and the benchmark [C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks[J]. Advances in neural information processing systems, 2012, 25.
Rao T, Li X, Xu M. Learning multi-level deep representations for image emotion classification[J].Neural processing letters, 2020, 51: 2043-2061.
Sowmyayani S, Rani P A J. Salient object based visual sentiment analysis by combining deep features and handcrafted features[J]. Multimedia Tools and Applications, 2022, 81(6): 7941-7955.
Li W, Liu K, Zhang L, et al. Object detection based on an adaptive attention mechanism[J].Scientific Reports,2020, 10 (1): 11307.
Hu J, Shen L, Sun G. Squeeze-and-excitation networks [C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.
Chen L, Zhang H, Xiao J, et al. Sca-cnn:Spatial and channel-wise attention in convolutional networks for image captioning [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 5659-5667.
Song K, Yao T, Ling Q, et al. Boosting image sentiment analysis with visual attention[J]. Neurocomputing,2018, 312: 218-228.
Yang J, She D, Lai Y K, et al. Weakly supervised coupled networks for visual sentiment analysis[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7584-7592.
Zhao S, Jia Z, Chen H, et al. PDANet: Polarity-consistent deep attention network for fine-grained visual emotion regression [C]// Proceedings of the 27th ACM international conference onmultimedia. 2019: 192-201.
Woo S, Park J, Lee J Y, et al. Cbam: Convolutionalblock attention module[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.
Zhang H, Xu D, Luo G, et al. Learning multi-level representations for affective image recognition[J]. Neural Computing and Applications, 2022, 34(16): 14107-14120.
Borth D, Ji R, Chen T, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]// Proceedings of the 21st ACM international conference on Multimedia. 2013: 223-232.
Rao T, Xu M, Liu H, et al. Multi-scale blocks based image emotion classification using multiple instance learning[C]// 2016 IEEE International Conference on ImageProcessing (ICIP). IEEE, 2016: 634-638.
You Q, Luo J, Jin H, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks [C]//Proceedings of the AAAI conference on Artificial Intelligence. 2015, 29(1).
Yang J, She D, Sun M, et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE Transactions on Multimedia, 2018, 20(9): 2513-2525.
Xiong H, Liu Q, Song S, et al. Region-based convolutional neural network using group sparse regularization for image sentiment classification[J]. EURASIP Journal on Image and Video Processing, 2019, 2019(1): 1-9.
Machajdik J, Hanbury A. Affective image classification using features inspired by psychology and art theory. Proceedings of the 18th ACM international conference on Multimedia, 2010. 1, 5.
