Bert-based model for Ceramic Emotional Text Recognition and Multi-label Classification

Busheng  Li; Siqi  Li; Wangren  Qiu; Tao Liu

doi:10.54097/kvtt3704

Authors

Busheng Li
Siqi Li
Wangren Qiu
Tao Liu

DOI:

https://doi.org/10.54097/kvtt3704

Keywords:

BERT model, multi-label classification, social media, sentiment analysis, deep learning

Abstract

Ceramic products are necessities for daily home life, while the emotional analysis research on ceramic products cannot keep up with the needs of social and economic development. To mine the valuable information on ceramic emotion quickly and effectively, this work has proposed a novel method for text recognition and classification of microblogging information related to Jingdezhen ceramics by fusing Bert model and multi-label classifiers on the novel dataset we first established. Firstly, the first multi-label emotion dataset consisting of 7564 samples was constructed on 10154 raw samples obtained with Python from microblogging information, which is noted as the keyword “Jingdezhen ceramics”. Secondly, the data were categorized with multiple labels into useful or useless information. Useful information was further classified into three categories: price information, appearance information, and quality information. Thirdly, a hybrid model combining Bert and multi-label classifiers has been provided for analyzing ceramic emotion. Based on a series of experiments, it has been observed that the predictor’s accuracy is 92% and its loss value is 0.33. This work highlights the annotation guidelines, dataset characteristics, and insights into different methodologies used for analyzing ceramic emotion and cultural heritage.

References

Kaliyar, R.K., A. Goswami, and P. Narang, FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multimedia tools and applications, 2021. 80(8): p. 11765-11788.

Yuan, L., et al., Transfer learning for hate speech detection in social media. arXiv preprint arXiv: 1906.03829, 2019.

Gundapu, S. and R. Mamidi, Transformer based automatic COVID-19 fake news detection system. arXiv preprint arXiv:2101.00180, 2021.

Li, Y., Application of Text Classification Technology in the Ceramic Industry. Modern Computer (Professional Edition), 2010(14): p. 60-63.

Yanan, Z., 'Moving with Emotion'—A Study on the Application of Emotional Elements in Daily Ceramic Design. 2013.

Yun, Y., et al., A Decorative Pattern Classification Model for Ancient Ceramics Based on Improved E-EfficientNet. Laboratory Research and Exploration, 2023. 42(02): p. 129-134+186.

Xinping, Y., Application of Data Mining in the Chronology of Ancient Ceramics and Classification of Ceramic Raw Materials. 2023.

Song, R., et al., Label prompt for multi-label text classification. Applied Intelligence, 2023. 53(8): p. 8761-8775.

Xiaofeng, L., L. Huili, and M. Yinglong, HE-HMTC: A hybrid embedding-based text representation for Hierarchical multi-label text classification. Software Impacts, 2022. 14.

Yuhao, Z., Research on Short Text Classification of News Based on BERT. 2021.

Chen, T., et al., The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 2020. 33: p. 15834-15846.

Wei, L., et al., Prompt Tuning for Multi-Label Text Classification: How to Link Exercises to Knowledge Concepts? Applied Sciences, 2022. 12(20): p. 10363.

Ming, S. and L. Yanlong, Application and Optimization of BERT in Sentiment Classification of Weibo Short Texts. Journal of Miniaturization for Technology 2021. 42(04): p. 714-718.

Luo, L., et al., An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics, 2018. 34(8): p. 1381-1388.

Zhang, T., et al., Revisiting few-sample BERT fine-tuning. arXiv preprint arXiv:2006.05987, 2020.

Chen, Z. and T. Qian. Transfer capsule network for aspect level sentiment classification. in Proceedings of the 57th annual meeting of the association for computational linguistics. 2019.

González-Carvajal, S. and E.C. Garrido-Merchán, Comparing BERT against traditional machine learning text classification. arXiv preprint arXiv:2005.13012, 2020.

Jeong, C., et al., A context-aware citation recommendation model with BERT and graph convolutional networks. Scientometrics, 2020. 124: p. 1907-1922.

Zeng, M., et al., Musicbert: Symbolic music understanding with large-scale pre-training. arXiv preprint arXiv:2106.05630, 2021.

Yanni, M., Research on Multi-label Sentiment Classification Problem Based on Machine Learning. Computer and Information Technology, 2023. 31(05): p. 12-15.

Wenbiao, Z., Research on Multi-label Classification Algorithm Based on Label Correlation Analysis. 2023.

Xuegang, L. and L. Junrui, A Water Quality Prediction Model Method Based on Tensor Feature-GRU and Multi-head Self-attention Mechanism. Journal of Panzhihua University, 2023. 40(05): p. 89-96.

Ashraf, N., et al., Multi-label emotion classification of Urdu tweets. PeerJ Computer Science, 2022. 8: p. e896.

Jian, D., L. Junhua, and C. Jingang, An Improved HiNT Text Retrieval Model Using BERT and Coverage Mechanism. Journal of Intelligent Systems: p. 1-8.

Huang, Y.-S. and Y.-H. Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. in Proceedings of the 28th ACM international conference on multimedia. 2020.

Hsiao, W.-Y., et al. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. in Proceedings of the AAAI Conference on Artificial Intelligence. 2021.

Lu, Z., P. Du, and J.-Y. Nie. VGCN-BERT: augmenting BERT with graph embedding for text classification. in Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part I 42. 2020. Springer.

Liu, Y., Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318, 2019.

Huang, W., et al., Toward fast and accurate neural chinese word segmentation with multi-criteria learning. arXiv preprint arXiv:1903.04190, 2019.

Goldberg, Y., Assessing BERT's syntactic abilities. arXiv preprint arXiv:1901.05287, 2019.

Kaneko, M. and M. Komachi, Multi-head multi-layer attention to deep language representations for grammatical error detection. Computación y Sistemas, 2019. 23(3): p. 883-891.

Delobelle, P., T. Winters, and B. Berendt, Robbert: a dutch roberta-based language model. arXiv preprint arXiv:2001.06286, 2020.

Devlin, J., et al., Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

Prasanna, S., A. Rogers, and A. Rumshisky, When BERT plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561, 2020.

Chuag, L., BERT's Application in Text Analysis. 2020.

Xie, S., et al., PALI at SemEval-2021 task 2: fine-tune XLM-RoBERTa for word in context disambiguation. arXiv preprint arXiv:2104.10375, 2021.

Bhasuran, B., BioBERT and Similar Approaches for Relation Extraction, in Biomedical Text Mining. 2022, Springer. p. 221-235.

Clark, K., et al., Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.

Lianxi, W., et al., Research on Recognition and Feature Distribution of ASEAN-related Public Opinion towards China: Analyzing Mainstream English and Chinese Media. Journal of Intelligence, 2022. 41(08): p. 94-101.

Joshi, M., et al., Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 2020. 8: p. 64-77.

Yang, Z., et al., Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 2019. 32.

Xu, W., et al. Symmetric regularization based bert for pair-wise semantic reasoning. in Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.

Chuan, C.-H. and D. Herremans. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based representation. in Proceedings of the AAAI Conference on Artificial Intelligence. 2018.

De Bruyn, M., et al. Bart for knowledge grounded conversations. in Proceedings of KDD Workshop on Conversational Systems Towards Mainstream Adoption (KDD Converse’20). ACM, New York, NY, USA. 2020.

McCarley, J., R. Chakravarti, and A. Sil, Structured pruning of a bert-based question answering model. arXiv preprint arXiv:1910.06360, 2019.