Real-Time Sign Language Translator

Abstract

Sign language is so widespread that people with hearing/speech impairments are familiar with it, and others are unfamiliar with it. As a result, there is a significant communication gap between people with speech and hearing impairments and the rest of the general population. A human sign language interpreter is a common solution for bridging this gap. However, because the number of sign language interpreters is small in comparison to the number of deaf and mute people in the world, some deaf and mute people cannot afford to use a human interpreter all of the time when communicating with others. This communication must be automated so that the deaf-mute community does not rely on human interpreters. This paper focuses on developing a system that can translate American Sign Language into words/sentences and vice versa in real-time, with some extra features that will help remove the communication barrier between ordinary and hearing/talking impaired people. The main function is to detect and identify sign language performed by the user. Initially, the system is trained to detect and identify signs using object detection and motion-tracking techniques. A convolutional neural network model was trained on a manually created ASL data set. The identified signs are then translated into English to form grammatically correct sentences. A text-to-text transformer built on an encoder-decoder architecture is used to detect grammatical errors and provide correct sentences. To further improve effectiveness, the system incorporates a feature to translate an image containing English text into American Sign Language. Furthermore, the system consists of a voice-to-sign language translator and a virtual sign keyboard. The methodology has been explained in further sections.