A Deep Feature–Based Dynamic Hand Gesture Recognition Framework for Word and Sentence Formation Using Mediapipe

Abstract

Individuals who are deaf or experience difficulties with hearing and speech predominantly rely on sign language as their medium to communicate, which is not universally comprehended leading to obstacles in achieving effective communication. Advances in deep learning technologies in recent years have enabled the development of systems intended to autonomously interpret gestures in sign language and translate them into spoken language. This paper introduces a system built on deep learning methodologies for recognizing sign language. It uses long short-term memory (LSTM) architecture to distinguish and classify hand gestures which are static and dynamic. The system is divided into three primary components, including dataset collection, neural network assessment, and sign detection component that encompasses hand gesture extraction and sign language classification. The project was divided into two major phases: Static Sign Recognition and Dynamic Sign Recognition. During the static phase, a fully original dataset comprising 120,000 images was created for 24 alphabets (A–Z excluding J and Z). Each image was then treated through two custom-designed filters: silhouette extraction and finger detection, carried out using MediaPipe for precise annotation and cropping on its own. The training of the static model happened with custom-designed CNN applied on the data to achieve accuracies of 94% (finger detection), 89% (silhouette), and 91% (combined). In the dynamic phase, signs with movement were captured using MediaPipe landmarks and registered as NumPy arrays (CSV and .NPY) files. Hand and facial keypoints were stored at every frame for temporal learning. This phase included one-hand, two-hand, and hand-with-face categories. The whole system achieved 15–20 FPS real-time inference on CPU alone. While the static signs and single-handed dynamic signs were performing very well, the accuracy of complex two-handed and face-assisted gestures suffered due to scarcity and noise in the data. The results provide a very good baseline to improve upon for the future using deep temporal networks like LSTM or 3D CNNs.