Performance Optimization and Interpretability Analysis of Lightweight Transformer Models for Speech Emotion Recognition Using Open-Source Emotional Speech Datasets

Abstract

Accurate speech-emotion recognition (SER) remains challenging across speakers, languages, and recording conditions, especially under tight computing conditions. We present a lightweight three-class SER framework that unifies acted corpora and pairs frozen self-supervised speech embeddings with a compact residual CNN. A Multi-Corpus Unified Labelling Protocol (MCULP) harmonises CREMA-D, RAVDESS, and EmoDB into a balanced three-class taxonomy (negative, sad_fear, pos_neutral), yielding 7,618 utterances with an 80/10/10 speaker-independent split. Our 3.9-M-parameter Residual Squeeze-and-Excitation 1-D CNN (ResAttn1D-CNN) uses five residual blocks with channel attention and 768×400 wav2vec 2.0-base embeddings. A Tri-Aug pipeline—Gaussian noise, random crop-pad, and SpecAugment-style temporal masking—improves robustness. Trained with AdamW and mixed precision, the model converges in ≈10 hours. On the held-out test set, it reaches 62.9% accuracy and 0.627 macro-F1, outperforming a strong three-layer CNN by 20.9 points and exceeding prior CNN results on comparable tasks. Ablations: +4.9 pp (attention), +2.9 pp (Tri-Aug), +3.2 pp (depth); the confusion matrix shows residual ambiguity between low-arousal negative and neutral speech. Real-time inference (<3.5 ms/utterance, <15 MB GPU) enables edge deployment. Code, manifests, and Docker recipes will be released for reproducibility and benchmarking.