Investigating the Impact of Parameter Variations of Transformer Models on Sentiment Classification

Abstract

With the development of the Internet, an increasing number of applications and websites appearing on the Internet allow movie viewers to add comments for movies. If people want to know what per cent of comments are positive or negative, it will cost a large amount human resources and time to check each comment. However, with the help of the transformer model, it will save a large number of human resources and time to finish sentiment classification for long movie comments. The dataset ‘IMDb’ used to train the transformer model is a large Movie Review Dataset for binary sentiment classification of movie reviews. Furthermore, since sentiment classification for movie comments does not require the decoder in the transformer model to predict the next token, the transformer model only need to preserve the part of Positional Encoding, Encoder and Multi-head self-attention mechanism. This paper will investigate how three parameters (the number of layers in encoder, the amount of heads in multi-head self-attention, expansion factor in position-wise fully connected feed-forward network) affect the performance of the transformer model and which set of parameters could allow the transformer to have the best performance. After researching on three parameters, the transformer model used to do sentiment classification for movie comments has the best performance when there are sixteen heads in multi-head self-attention mechanism, four layers in Encoder, and expansion factor in feed-forward network which is position-wise fully connected is four.