An Analysis of Knowledge Distillation Methods for Sentiment Analysis: Limitations and Future Prospects

Abstract

With the advancement of sentiment analysis, the integration of multi-teacher and multi-modal models has grown increasingly prominent. However, the direct fusion of these diverse models poses significant challenges. This paper presents a comprehensive review of knowledge distillation techniques designed to address these challenges. This paper systematically surveys several key approaches: for multi-teacher scenarios, this paper examines ensemble knowledge distillation (EKD) as a method for effective model compression; for multi-modal applications, this paper explores cross-modal distillation (CMD) and graph-based distillation to effectively handle the intrinsic heterogeneity between data modalities. Besides, this paper analysis identifies critical limitations in current methods, including the static nature of knowledge extraction and constraints of graph distillation approaches. To address these shortcomings, this paper proposes several forward-looking research directions, such as incorporating meta-learning for dynamic weight adjustment, using attention mechanisms for adaptive teacher selection, and replacing conventional Graph Neural Networks (GNNs) with Hypergraph Neural Networks (HNNs) to capture higher-order inter-modal relationships. By critically examining existing methods and a clear outlook on future trends, this paper provides valuable insights and a roadmap for researchers in the field.