This project is called “Bridge: an AI powered sign language interpreter”. This project intends to create a prototype ****that can translate International Sign Language into English. Considering past studies and projects, this project contributes to continuous discourse regarding research in the most efficient and accurate machine learning methods for training a model that can classify real-time movements.
Several studies have tackled the same goal of enhancing the communication between sign language users and verbal speakers. Some have considered comparisons between methods, some about models and some have introduced new ideas about UI/UX.
In a study by Johnny and Nirmala, they tried to measure the efficiency of different machine learning methods in trying to classify sign language inputted through a technology called Fifth Dimension Technologies (5DT) gloves (2022). They compared using Neural Networks, k-NN algorithm and Decision Tree Classifier in terms of accuracy. They discovered that for smaller datasets, k-NN algorithm is the most accurate and applicable for various testing situations. For the same situation, Neural Networks are not as effective as there needs to be diverse data to prevent overfitting. All the while Decision Tree Classifier is deemed unusable.
5DT gloves are gloves often used for Virtual Reality purposes
This study is important to give important considerations in choosing a proper algorithm or training method. This validates my original idea of using Neural Networks to train my own model. Although there are similarities in mine and this studies goals, one of my main focuses is to purely use a portable device as inputs for the model. In addition, it should be purely just a software in a device as to make it convenient and making it accessible to the users. This study uses a specific glove that maps the location of the users’ hands. This input is used to create a classification. This study claims to have an accuracy of almost 97% with k-NN, however, a disclaimer was given that for a scalable model, their very high accuracy is not applicable. Having the goal of making this project continue to be scalable, using this k-NN would probably be not beneficial. Despite this, it is worth looking over.
Another study by Eryilmaz explores a similar idea, the author aimed to provide a comprehensive understanding of machine learning models used in sign language recognition. The article focuses on evaluating different models and analyzing their performance in recognizing sign language gestures. The results of the experiments indicate that the hybrid models achieve the highest accuracy in sign language recognition, outperforming both CNN-based and LSTM-based models. This study also provides insights into the factors that contribute to the performance differences among the models, such as the complexity of sign language gestures and the limitations of the datasets. An important consideration that was introduced was the similarity of the signs along with their meanings, this is caused by a specificity by the Turkish sign language. This does its training through pre-recorded videos instead of training it on the go.
This article is relevant to the research project on creating an AI-powered sign language interpreter. It offers valuable information on the effectiveness of different machine learning models in recognizing sign language gestures. The findings can inform the selection and development of the most suitable model for the project, considering factors such as accuracy and computational efficiency. I originally wanted to use an LSTM-based model; however, this study shows that a hybrid model is possible. Knowing that only an LSTM is the focus of this study, I want to explore more sequence-based data to train the model such as GRU or ARIMA. In addition, about their methodology, they used pre-recorded videos, which is different from my original plan of using myself to train the model. I think there are a lot of benefits to using videos. This makes it possible to train it with a high volume of datasets instead of doing everything by one. I will consider using this methodology as well, but this necessitates important logistics to record from different sources.
SignAll is a company that specializes in the translation of real time video for sign language users. The SignAll SDK utilizes MediaPipe, a framework for building multimodal applied ML pipelines, to provide real-time sign language recognition capabilities. The article highlights the importance of accessibility and inclusivity in technology and emphasizes the potential of the Signal SDK to empower sign language users. The Signal SDK uses a lot of features, including hand tracking, hand landmark estimation, and gesture recognition. It enables developers to build applications that can accurately interpret sign language gestures and translate them into text or other forms of communication. The SDK supports various sign languages and can be customized to suit specific user needs. The article also discusses the technical aspects of the Signal SDK, such as the underlying machine learning models and the integration with MediaPipe.
SignAll SDK User interface
In terms of creating a commercialized product, SignAll might be the closest one. They use several ways to convert a video stream into the translator. They use multiple cameras and depth sensors to create a very accurate model. They have admitted issues in transferring to devices that anyone can access as it is an important compromise between accuracy and convenience, however, the most interesting part that they have achieved in relation to this study is their UI. Considering sign language users will put their phones far away from themselves, SignAll has made it possible to use your fingers as a pointer to select different functions. As seen in the photo above, there are buttons available for choosing words and a keyboard. I will definitely look into this and try to ask more from potential users what type of UI would best fit their needs.