Research on Optimising Data Detection Timing and Anomaly Detection Using K-Means Clustering and Random Forests

Abstract

 This study integrates Pearson correlation analysis, multiple linear regression, linear mixed models (LMM), K-means clustering, and random forest algorithms to construct a data-driven framework for detection optimisation and anomaly identification. It aims to enhance sequencing data processing efficiency and improve the accuracy of anomaly detection. During data preprocessing, invalid and outlier values were first removed, followed by feature standardisation. Pearson correlation analysis was employed to uncover associations among core features. After constructing a multiple linear regression model using the least squares method, random effects were introduced to optimise the model into an LMM to address its insufficient fit. This approach not only significantly reduced the AIC and BIC values but also decreased the prediction error (RMSE) by 22%, substantially improving model fit and predictive accuracy. To further optimise detection timing, K-means clustering divided key influencing factors into five reasonable intervals. The median method was employed to determine optimal detection points within each interval, enabling earlier detection while maintaining accuracy. For anomaly detection requirements, the dataset was refined through data partitioning, mean imputation of missing values, and feature engineering. The resulting random forest model achieved 98% accuracy on the test set, demonstrating balanced precision and recall while effectively identifying core influential features. This integrated framework achieves seamless coordination between sequencing data processing, detection timing optimisation, and anomaly detection through multi-algorithm collaboration and iterative model refinement, providing a robust technical solution for related detection tasks.