Study on real-time gesture recognition using convolutional neural network based on SMOTE method

Pengyu Lin

doi:10.54097/awr58j94

Authors

Pengyu Lin

DOI:

https://doi.org/10.54097/awr58j94

Keywords:

CNN, gesture recognition, smote, mediapipe.

Abstract

Today, there are still many deaf individuals in the world, and enabling barrier-free communication for them remains a global challenge. This study aims to investigate real-time gesture recognition using convolutional neural networks and the mediapipe library, addressing data imbalance through the SMOTE method. In this paper, we investigate the developmental trajectory, underlying principles, and multifaceted applications of sign language recognition technology across various domains. We also introduce a sign language recognition system utilizing a deep learning algorithm to improve accuracy and stability, thereby contributing to the continuous progress of sign language recognition technology. The results demonstrate that SMOTE effectively enhances the experimental data, while mediapipe plays a significant role in improving recognition accuracy. The proposed method shows promise in real-time gesture recognition for the deaf, offering a viable solution to enhance their communication. This research contributes to the field of assistive technologies for individuals with disabilities and has potential for further optimization and real-world applications.

Downloads

Download data is not yet available.

References

[1] Asadi-Aghbolaghi M, Clapés A, Bellantonio M, et al. Deep learning for action and gesture recognition in image sequences: A survey. Gesture Recognition, 539-578 (2017).

[2] Ferreira P M, Cardoso J S, Rebelo A. On the role of multimodal learning in the recognition of sign language. Multimedia Tools and Applications, 78 (8): 10035-1005 (2009).

[3] Guo X L, Yang T T. Gesture recognition based on HMM-FNN model using a Kinect. Journal on Multimodal User Interfaces, 11: 1-7 (2017).

[4] Peng Y, Lin M, Chen Y, et al. Personal Touch: Improving touchscreen usability by personalizing accessibility settings based on individual user's touchscreen interaction, 2019 CHI Conference on Human Factors in Computing Systems, 1-11 (2019).

[5] Oberweger M, Wohlhart P, Lepetit V. Hands deep in deep learning for hand pose estimation. arXiv preprint arXiv:1502.06807 (2015).

[6] Rastgoo R, Kiani K, Escalera S. Multi-modal deep hand sign language recognition in still images using restricted Boltzmann machine. Entropy, 20 (11): 809 (2018).

[7] Zhang Q, Yang M, Zheng Q, et al. Segmentation of hand gesture based on dark channel prior in projector-camera system, 2017 IEEE/CIC International Conference on Communications in China (ICCC). IEEE, 1-6 (2017).

[8] Wadhawan A, Kumar P. Deep learning-based sign language recognition system for static signs. Neural computing and applications, 32 (12): 7957-7968 (2020).

[9] Bilaniuk L T, Atlas S W, Zimmerman R A. Magnetic resonance imaging of the orbit. Radiologic Clinics of North America, 25 (3): 509-528 (1987).

[10] Elreedy D, Atiya A F, Kamalov F. A theoretical distribution analysis of synthetic minority oversampling technique (SMOTE) for imbalanced learning. Machine Learning, 1-21 (2023)