A Comparative Exploration of BERT, RNN, and GRU for Sentiment Classification
DOI:
https://doi.org/10.54097/3v5crt36Keywords:
Sentiment classification, deep learning, recurrent neural network, gated recurrent unit.Abstract
Sentiment classification tasks are a large branch of current natural language processing problems. The purpose of this task lies in determining the sentiment that the text is trying to convey given a piece of text or sentence. This technique can be widely referenced in areas such as, monitoring market opinion, protecting social security, and developing ai customer service. Currently, the most advanced solutions to the sentiment classification problem are based on Bidirectional Encoder Representations from Transformers (BERT), in order to try to research a better solution, this study will start from the bottom, first find the optimal Recurrent Neural Network (RNN) model to pave the way for future improvements. The research method is to start from one layer and keep adjusting the parameters or RNN types until the best model is found. Through an experimental study, it was found that the two-layer bidirectional Long Short-Term Memory (LSTM) performed best on the sentiment classification problem with an accuracy of 93.21%. It proves that LSTM is better than Gated Recurrent Unit (GRU) and simple RNN in understanding long text, and in the future, more datasets can be collected from other platforms, not twitter, and then the model can be trained more complexly, which will help the model to be generalized to other domains or occasions.
Downloads
References
[1] Devlin Jacob, Chang Ming-Wei, Lee Kenton, et al. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint, 2018: 1810.04805.
[2] Liu Yinhan, Ott Myle, Goyal Naman, et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint, 2019: 1907.11692.
[3] Sanh Victor, Debut Lysandre, Chaumond Julien, et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint, 2019: 1910.01108.
[4] Nidula Elgiriyewithana, Emotions Dataset, Kaggle, URL: https://www.kaggle.com/datasets/nelgiriyewithana/emotions. Last Accessed: 2024/08/02
[5] Chen Wuxing, Yang Kaixiang, Yu Zhiwen, et al. A survey on imbalanced learning: latest research, applications and future directions. Artificial Intelligence Review, 2024, 57 (6): 1-51.
[6] Shahid Farah, Aneela Zameer, Muhammad Muneeb. Predictions for COVID-19 with deep learning models of LSTM, GRU and Bi-LSTM. Chaos, Solitons & Fractals, 2020, 140: 110212.
[7] Sokolova Marina, Guy Lapalme. A systematic analysis of performance measures for classification tasks. Information processing & management, 2009, 45 (4): 427-437.
[8] Minaee Shervin, Kalchbrenner Nal, Cambria Erik, et al. Deep learning--based text classification: a comprehensive review. ACM computing surveys, 2021, 54 (3): 1-40.
[9] Cahuantzi Roberto, Xinye Chen, Stefan Güttel. A comparison of LSTM and GRU networks for learning symbolic sequences. Science and Information Conference, 2023: 771-785.
[10] Maurya, Manju, Neha Yadav. A comparative analysis of gradient-based optimization methods for machine learning problems. International Conference on Data Analytics and Computing, 2022: 85-102.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Highlights in Science, Engineering and Technology

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.







