Parameter Optimization in DeBERTa for Text Classification Using PCA on Limited GPU Resources

Abstract

This study focuses on optimizing Decoding-enhanced Bidirectional Encoder Representations from Transformers with disentangled Attention (DeBERTa) for text classification tasks, particularly on smaller datasets with limited GPU resources, by using Principal Component Analysis (PCA) to adjust and explain parameter choices. The goal is to reduce the model's parameters without significantly compromising accuracy. Specifically, the research employs DeBERTav3 on the Internet Movie Database (IMDb) dataset, without pre-training, and uses PCA to analyse the embedding and attention components of the model. PCA is applied to identify and reduce parameters by observing the Number of smallest factors that has a given Neglected cumulative contributing Rate of principal components (NNR), iteratively refining the model until optimal efficiency is achieved. This study demonstrates that PCA can effectively guide parameter adjustments, allowing even a smaller model to maintain high accuracy. The IMDb dataset is used to validate the approach, showing that a reduced DeBERTa model can still achieve competitive performance, making it particularly useful for natural language processing tasks involving minority languages or constrained computational environments. The findings have broader implications for optimizing large language models, suggesting that future work could explore combining PCA with techniques like factorized embedding parameterization to further enhance efficiency.