Multimodal Depression Recognition Based on Sentence-level Dynamic Multimodal Split Attention Fusion
DOI:
https://doi.org/10.54097/zs7j8602Keywords:
Depression, Multimodal, Sentence-level Dynamic Multimodal Split Attention FusionAbstract
Depression is a common yet highly covert mental disorder, making the development of efficient intelligent recognition methods crucial for early screening and clinical diagnostic support. Existing multimodal depression recognition approaches still face limitations in modal interaction and long-sequence semantic modeling, struggling to fully capture local dynamics and cross-modal dependencies. To address this, this study proposes a multimodal temporal fusion network. This approach first divides long medical interview sequences into sentence-level units based on timestamps to mitigate information dilution in lengthy sequences. Subsequently, it designs a sentence-level dynamic multimodal attention fusion module. This module further segments sentence sequences into contiguous segments and adaptively emphasizes key modal features while suppressing redundant and noisy information through dynamic weight allocation. On the public dataset DAIC-WOZ and the self-built Chinese dataset MDD2025, MTFNet achieves accuracy rates of 86% and 84%, respectively.
Downloads
References
[1] Depression WHO. Other common mental disorders: global health estimates[J]. Geneva: World Health Organization, 2017, 24(1).
[2] Wittchen H U, Jacobi F, Rehm J, et al. The size and burden of mental disorders and other disorders of the brain in Europe 2010[J]. European neuropsychopharmacology, 2011, 21(9): 655-679.
[3] Lim S S, Vos T, Flaxman A D, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010[J]. The lancet, 2012, 380(9859): 2224-2260.
[4] Mathers C D, Loncar D. Projections of global mortality and burden of disease from 2002 to 2030[J]. PLoS medicine, 2006, 3(11): e442.
[5] World Health Organization. The global burden of disease: 2004 update [M]. Geneva: World Health Organization, 2008.
[6] Li J Y, Li J, Liang J H, et al. Depressive symptoms among children and adolescents in China: a systematic review and meta-analysis[J]. Medical Science Monitor, 2019, 25: 7459–7470. DOI: 10.12659/MSM.916774.
[7] Marcus S M, Flynn H A, Blow F C, et al. Depressive symptoms among pregnant women screened in obstetrics settings[J]. Journal of Women's Health, 2003, 12(4): 373–380. DOI: 10.1089/154099903765448886.
[8] Grace S L, Evindar A, Stewart D E. The effect of postpartum depression on child cognitive development and behavior: A review and critical analysis of the literature[J]. Archives of Women's Mental Health, 2003, 6(4): 263–274. DOI: 10.1007/s00737-003-0024-6.
[9] Zenebe Y, Akele B, W/Selassie M, Necho M. Prevalence and determinants of depression among old age: a systematic review and meta-analysis[J]. Annals of General Psychiatry, 2021, 20: 55. DOI: 10.1186/s12991-021-00375-x.
[10] Kim A Y, Jang E H, Lee S H, et al. Automatic depression detection using smartphone-based text-dependent speech signals: deep convolutional neural network approach[J]. Journal of medical Internet research, 2023, 25: e34474.
[11] Amanat A, Rizwan M, Javed A R, et al. Deep learning for depression detection from textual data[J]. Electronics, 2022, 11(5): 676.
[12] Mahayossanunt Y, Nupairoj N, Hemrungrojn S, et al. Explainable depression detection based on facial expression using LSTM on attentional intermediate feature fusion with label Smoothing[J]. Sensors, 2023, 23(23): 9402.
[13] Zhang G, Zhuo G, Yang Y, et al. Sentence-level multi-modal feature learning for depression recognition[J]. Frontiers in Psychiatry, 2025, 16: 1439577.
[14] Ye J, Yu Y, Wang Q, et al. Multi-modal depression detection based on emotional audio and evaluation text[J]. Journal of Affective Disorders, 2021, 295: 904-913.
[15] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural computation, 1997, 9(8): 1735-1780.
[16] Liu G, Guo J. Bidirectional LSTM with attention mechanism and convolutional layer for text classification[J]. Neurocomputing, 2019, 337: 325-338.
[17] Lang S, Chuqing H, Guofa L, et al. MSAF: multimodal split attention fusion[J]. CoRR, 2020.
[18] A K K , B T W S , C R L S ,et al. The PHQ-8 as a measure of current depression in the general population[J].Journal of Affective Disorders, 2009, 114( 1–3):163-173.DOI: 10. 1016/ j.jad.2008.06.026.
[19] SHEN Y, YANG H, LIN L. Automatic depression detection: An emotional audio-textual corpus and a gru/bilstm-based model; proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), F, 2022 [C]. IEEE.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
