Visual Insights in Lung Cancer Prediction: Leveraging Data Visualization and Analysis for Building and Tuning Machine Learning Models

Abstract

Lung cancer stands as a significant global health challenge with its high occurrence rates and fatal lethality. Given lung cancer's severity and complicated symptoms in later stages, healthcare facilities must detect them early. Traditional diagnostic methodologies are often burdened by high cost and specialized expertise, prompting exploration of alternative methods, such as machine Learning. This paper utilizes a dataset from a Nature Medicine study involving 462,000 participants in China and employs machine learning to predict lung cancer risk. Through the application of Exploratory Data Analysis (EDA), this paper explored the correlation between demographic variables, environmental factors, and lifestyle habits with the probability of lung cancer occurrence. The neural network architecture incorporates dynamic layers, rectified linear unit (ReLU) activation, and the Adam optimizer with dropout regularization. Results from EDA reveal correlations between lifestyle factors and lung cancer risk. The machine learning model achieved 70% accuracy on predictions and was refined to 90% through EDA and utilizing the LIME interpreter. This study aimed to advance lung cancer prediction by applying an amalgamation of techniques and technologies, like data visualization, extensive data analysis, and machine learning. This research aimed to contribute to the showcase of applying the latest technological advancements for the era of big data within the medical research and healthcare industry. The success of this model suggests the viability of creating machine learning models targeted for cancer predictions and indicates further advancements, such as personalized prediction models.