Discovery of Weibo New Words Based on Rules and SVM

Abstract

 This article combines the proposed word features with SVM for Weibo new word recognition and extraction. Firstly, by modifying the segmentation dictionary to simulate Weibo new words, the training and testing corpus are segmented using the segmentation dictionary, and various proposed word features are counted. Then, the positive and negative samples extracted from the training corpus are vectorized using word features, and different kernel functions are selected to obtain Weibo new word classification support vectors through SVM training. By adding relaxation variables to improve the accuracy of classification, the Weibo new word classification support vector obtained from the training corpus and the Weibo new word candidate vector obtained from the test corpus are combined for SVM testing to obtain the calculated value of each candidate Weibo new word. The final Weibo new word recognition result is obtained by comparing the calculated value and threshold. Experiments have shown that the combination of word features and SVM can be used for the recognition and extraction of Weibo new words, and relatively good results have been achieved. This method can be extended to the application field of Weibo new word recognition.