Improved Text Matching Model Based on BERT
DOI:
https://doi.org/10.54097/fcis.v2i3.5209Keywords:
Bert, BiLSTM, CNN, Text Matching, NLPAbstract
Text matching is a basic and important task in natural language understanding, this paper proposes a new model BBMC for the problem of insufficient feature extraction ability of existing text matching models, which integrates BiLSTM and multi-scale CNN on the basis of BERT. First, the word embedding representation of the text is obtained by the BERT, and then the semantic features of the text are further extracted by the double-layer BiLSTM, followed by the multi-scale CNN model, the key local features are extracted, and finally the linear and SoftMax function are used to classify. Experimental results on the LCQMC dataset show that the BBMC has been improved to a certain extent compared with other methods, and the accuracy on the test set can be best achieved 88.01%.
Downloads
References
Pang Liang, Lan Yanyan, Xu Jun, Guo Jiafeng, Wan Shengxian, Cheng Xueqi. Overview of Deep Text Matching [J]. Journal of Computer Science, 2017,40 (04): 985-1003.
Yang R, Zhang J, Gao X, Ji F, Chen H. Simple and effective text matching with richer alignment features[J]. arXiv preprint arXiv:1908.00300, 2019.
Huang P S, He X, Gao J, Deng L, Acero A, Heck L. Learning deep structured semantic models for web search using clickthrough data[C]//Proceedings of the 22nd ACM international conference on Information & Knowledge Management. 2013: 2333-2338.
Shen Y, He X, Gao J, Deng L, Mesnil G. A latent semantic model with convolutional-pooling structure for information retrieval[C]//Proceedings of the 23rd ACM international conference on conference on information and knowledge management. 2014: 101-110.
Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, et al. Semantic modelling with long-short-term memory for information retrieval[J]. arXiv preprint arXiv:1412.6629, 2014.
Hu B, Lu Z, Li H, Chen Q. Convolutional neural network architectures for matching natural language sentences[J]. Advances in neural information processing systems, 2014, 27.
Yin W, Schütze H, Xiang B, Zhou B. Abcnn: Attention-based convolutional neural network for modeling sentence pairs[J]. Transactions of the Association for Computational Linguistics, 2016, 4: 259-272.
Pang L, Lan Y, Guo J, Xu J, Wan S, Cheng X. Text matching as image recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2016, 30(1).
Chen Q, Zhu X, Ling Z, Wei S, Jiang H, Inkpen D. Enhanced LSTM for natural language inference[J]. arXiv preprint arXiv:1609.06038, 2016.
Graves A, Schmidhuber J. Framewise phoneme classification with bidirectional LSTM networks[C]//Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. IEEE, 4: 2047-2052.
Wang Z, Hamza W, Florian R. Bilateral multi-perspective matching for natural language sentences[J]. arXiv preprint arXiv:1702.03814, 2017.
Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality[J]. Advances in neural information processing systems, 2013, 26.
Pennington J, Socher R, Manning C D. Glove: Global vectors for word representation[C]//Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014: 1532-1543.
Devlin J, Chang M W, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.
Li B, Zhou H, He J, et al. On the Sentence Embeddings from Pre-trained Language Models[C]// Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
Xia T, Wang Y, Tian Y, Chang Y. Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks[J]. 2021.
Meng Jinxu, Shan Hongtao, Wan Junjie, Jia Renxiang. BSLA: Improved Text Similarity Model of Siamese LSTM [J]. Computer Engineering and Application. 2021.
Cui Y, W Che, T Liu, B Qin, Z Yang, S Wang and G Hu, “Pretraining with whole word masking for chinese bert,” arXiv preprint arXiv:1906.08101, 2019.
Cui Y, Che W, Liu T, Qin B, Yang Z. Pre-training with whole word masking for chinese bert[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3504-3514.
Liu X, Chen Q, Deng C, Zeng H, Chen J, Li D, et al. Lcqmc: A large-scale chinese question matching corpus[C]//Proceedings of the 27th International Conference on Computational Linguistics. 2018: 1952-1962.
Zhang Wenhui, Wang Meiling, Hou Zhirong. A short text matching model incorporating contextual semantic differences [J/OL]. Journal of Peking University (Natural Science Edition) https://doi.org/10.13209/j.0479-8023.2022.071.
Downloads
Published
Issue
Section
How to Cite
