Study on Small Sample Text Classification Based on Multi-Level Self-Attention and Multi-Feature Residual Fusion under Data Enhancement

Abstract

In commercial applications, while traditional models can achieve comparable performance to mainstream large language models, they generally necessitate extensive training data. This requirement presents a significant challenge when processing complex, lengthy Chinese texts and multi-label classification tasks with limited data availability. Furthermore, conventional data augmentation techniques frequently disrupt the original word order, thereby diminishing their efficacy for pre-trained language model applications. To overcome these limitations, we introduce the MacBERT-CNN-BiLSTM model, which incorporates a multi-level self-attention mechanism to dynamically weight the integrated features extracted from MacBERT, CNN, and BiLSTM components. Our methodology preserves the integrity of original features during the final fusion phase by combining weighted features with original features through residual connections, thus generating a comprehensive final representation. This approach culminates in our MacBERT-BiLSTM-CNN-ResAttNet model (MBCResAttNet), specifically designed for multi-label classification of small-sample Chinese literature abstracts. We conducted extensive evaluations of our model across three datasets: AEDA-augmented, EDA-augmented, and original samples, benchmarking against six alternative models. The empirical results demonstrate that incorporating pre-trained language models substantially enhances classification performance. Moreover, the multi-level self-attention mechanism combined with residual feature fusion effectively captures global textual patterns, resulting in significant performance improvements. In the context of pre-trained language models, AEDA demonstrates superior efficacy compared to EDA in maintaining original semantic integrity. Additionally, the residual feature fusion methodology preserves critical original information while markedly improving model performance. With the implementation of AEDA augmentation, all evaluated models exhibited performance gains exceeding 10%, with our MBCResAttNet model attaining 96.17% accuracy—representing a substantial 13.41% improvement over baseline methods.