The Text Classification Method Based on BiLSTM and Multi-Scale CNN

Bo  He; Yongfen  Yang; Lei Wang; Jingxuan  Zhou

doi:10.54097/ypxxse31

Authors

Bo He
Yongfen Yang
Lei Wang
Jingxuan Zhou

DOI:

https://doi.org/10.54097/ypxxse31

Keywords:

Text classification; Neural networks; Deep learning; Natural language processing.

Abstract

Text classification is an important fundamental technique for efficiently managing the huge amount of textual information on the Internet. In the past few years, due to the unprecedented success of deep learning research, the task of text categorization has gradually become the center of gravity in the field of natural language processing, and the algorithms applied to it have also transitioned over time from traditional machine learning methods to neural network models based on deep learning, as well as the emerging attention mechanisms and pre-trained language models with good results. This paper outlines the traditional machine learning methods and deep learning methods involved in text classification, comprehensively analyzes the research progress and achievements of deep learning models for text classification, and compares the advantages and disadvantages of each. Then the evaluation metrics used for text classification and commonly used labeled datasets are introduced. Finally, the challenges faced by the text classification task and the difficulties to be further researched are summarized and outlooked to provide reference and support for future researchers in this field.

References

Tiejun Zhao, Muven Xu, Antony Chen. A review of natural language processing research[J]. Journal of Xinjiang Normal University (Philosophy and Social Science Edition), 2023,1-23.

WU SH,CHEN S P.Spam message recognition based on TFIDF and Self-Attention-Based Bi-LSTM[J].Computer Application Systems,2020,29(9):171-177.

GAN C Q, FENG Q D, ZHANG Z F. Scalable multi-channel dilated CNN-BiLSTM model with attention mechanism for Chinese textual sentiment analysis[J]. Future Generation Computer Systems, 2021, 118(1): 297-309.

ROY A M. An efficient multi-scale CNN model with intrinsic feature integration for motor imagery EEG subject classification in brain-machine interfaces[J]. Biomedical Signal Processing and Control, 2022, 74(4): 1-14.

Ming Li,Ying Li,Wangqin Lou,Lisheng Chen. A hybrid recommendation system for Q&A documents[J].Expert Systems With Applications,2022.

Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2023. Personalized News Recommendation: Methods and Challenges. ACM Transactions on Information Systems 41, 1 (2023), 1–50.

Cortes C, Vapnik V. Support-Vector Networks J . Machine Learning,1995,20(3):273-297.

SONG Zhongshan,NIU Yue,ZHENG Lu, et al. Multiscale CNN convolutional and global relation for Chinese text classification model[J]. Computer Engineering and Applications,2023 ,59(20):103-110.

KIM Y.Convolutional neural networks for sentence classifification[J].arXiv:1408.5882,2014.

CONNEAU A, SCHWENK H,BARRAULT L,et al.Verydeep convolutional networks for text classification [J].arXiv:1606.01781,2016.

LE H T,CERISARA C,DENIS A.Do convolutional networks need to be deep for text classification[J]. arXiv: 1707.04108,2017.

JOHNSON R,ZHANG T.Deep pyramid convolutional neu-ral networks for text categorization[C]//Proceedings of the55th Annual Meeting of the Association for Computa-tional Linguistics(Volume 1:Long Papers),2017:562-570.

GUO J,YUE B,XU G,et al.An enhanced convolutionalneural network model for answer selection[C]//Proceed-ings of the 26th International Conference on World WideWeb Companion,2017:789-790.

WANG H,HE J,ZHANG X,et al.A short text classification method based on n-gram and CNN[l].Chinese Jour-nal of Electronics,2020,29(2):248-254.

Adams B., McKenzie G.Crowdsourcing the character of a place: character-levelconvolutional networks formultilingual geographic text classification[J]. Transactionsin Gis, 2018, 22(2): 394-408.

Chen Zhuang, Qian Tieyun. Transfer capsule network for aspect level sentimentclassification[C]//Proceedings of the 57th Annual Meeting of the Association forComputational Linguistics. Florence: ACL press, 2019: 547-556.

Guo Bao, Zhang Chunxia, Liu Junmin, et al. Improving text classification withweighted word embeddings via a multi-channel TextCNN model[,. Neurocomputing.2019,363:366-374.

HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural computation,1997,9(8):1735-1780.

SCHUSTER M, PALIWAL K K. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 1997, 45(11):2673-2681.

ZHOU P,QI Z,ZHENG S,et al.Text classification improvedby integrating bidirectional LSTM with two-dimensionalmax pooling[J].arXiv:1611.06639,2016.

HUANG JI,LIN JO,HE YJ,et al.Chinese shorttext classification algorithm based on local semantics andcontext [1].Computer Engineering and Applications, 2021,57(6):94-100.

XU XK,ZHOU Z YA multi-scale BILSTMCNNbased emotion classification model for Wechat tweetsand its application [J].Information Science, 2021 , 39(5):130-137.

VELICKOVIC P,CUCURULL G,CASANOVA A,et al.Graph attention networks[Jl.arXiv:1710.10903,2017

XU B B,CEN K Y,HUANG J,et al.A survey on graphconvolutional neural network[I].Chinese Journal of Com-puters,2020,43(5):755-780.

Li R,Chen H,Feng F,et al.Dualgraph convolutiona networks foraspect-basedanalysis[C]sentimentonline:Proceedings of the Fifty-ninth Annual Meeting ofthe Association for Computational Linguistics and theEleventh International Joint Conference on NaturalLanguage Processing,2021.

Shen w,Wu S,Yang Y,et al.Directed acyclic graph ne-tworkfor conversational emotion recognition[J/OL].(2021-5-27)[2022-9-10]

Yan H,Gui L,Pergola G,et al.Position bias mitigation: Aknowledge-aware graph model for emotioncauseextraction [J/OL].(2021-6-7)[2022-9-10]

YANG T,HU L,SHI C,et al.HGAT:heterogeneous graphattention networks for semi-supervised short text classi-fication[1]. ACM Transactions on Information Systems, 2021,39(3):32.

KENTON J D M-W C, TOUTANOVA L, K. BERT: pre - training of deep bidirectional transformers for language understanding IC].Proceedings of NAACL-HLT,2019: 4171-4186.

DAI Z, YANG Z, YANG Y, et al. Transformer- XL: attentive language models beyond a fixed -length context [C]Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 2978-2988.

Cui Yiming, Che Wanxiang, Liu Ting, et al. Pre-training with whole word masking for chinese bert[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022,29: 3504-3514.

Kong Jun, Wang Jin, Zhang Xuejie. Hierarchical bert with an adaptive fine-tuningstrategy for document classification[J]. Knowledge-Based Systems, 2022,238:107872.

Jindian Su, Shanshan Yu, Xiaobin Hong. A self-supervised pretraining method for Chinese spelling error correction[J]. Journal of South China University of Technology (Natural Science Edition), 2023, 51 (09): 90-98.

Li Biang, Ma Hongchao,Zhou Qinglei. Sarcasm detection based on transfer learning[J]. Computer Application Research, 2021.38(12):3646-3650.

Sun Xiaoyan,Qiao Yali. False comment recognition gate based on migration and semi-supervised symbiotic fusion. Journal of Nanjing University (Natural Science),2022.58(05):846-855.

Soni S,Chouhan SS,Rathore SS.TextConvo Net: Aconvolutional neural network based architecture for textclassification [J/OL]. (2022-3-10)[2022-9-10]

ZHANG X,ZHAO J,LECUN YCharacter-level convolutional networks for text classification[C]//Advances inNeural Information Processing Systems,2015,28:649-657

WANG J,WANG Z,ZHANG D,et al.Combining knowl-edge with deep convolutional neural networks for shorttext classification[C]//26th International Joint Conferenceon Artificial Intelligence,2017:2915-2921.

WU C Y,DIAO Q,QIU M,et al.Jointly modeling aspects,ratings and sentiments for movie recommendationC]// Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014:193-202.

SOCHER R,PERELYGIN A,WU J,et al.Recursive deepmodels for semantic compositionality over a sentimentreebank [C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,2013:1631-1642

YAO L,MAO C,LUO YGraph convolutional networksfor text classification[C]//Proceedings of the 33rd AAAIConference on Artificial Intelligence,2019:7370-7377

PHAN X H,NGUYEN L M,HORIGUCHI S.Learning toclassify short and sparse text & web with hidden topics firom large-scale data collections[C]//Proceedings of the17th International Conference on World Wide Web,2008 :91-100.

LI X,ROTH D.Learning question classifiers[C]//19th International Conference on Computational Linguistics ,2002

PANG B,LEE L.Seeing stars:exploiting class relationships for sentiment categorization with respect to rating scales[J].arXiv:cs/0506075 ,2005.