My project is titled Bridge, it is an AI powered sign language interpreter and as the week progresses, I have been considering a lot of the technical directions that I will take moving into the project. Along with this, I have started to communicate with a mentor that might help me improve the considerations of my study.

Possible Hardware and Software considerations

As I continue to delve into research, I have taken inspiration from the article I saw last week in google developers. As much as I want to simplify the project, I think a part of the accuracy that they have achieved is augmented by the sensors that they had. They primarily had depth sensors and extra cameras to consider things that are covered by the body when they put their limbs in front of other parts.

I was thinking of a possible and simple way to implement this as well, this is why I got the idea of using the Lidar sensor in the iPad, however, the iPad I have is not the pro version so it does not have this accessory, it would have been really useful to see the possible optimizations I could implement with this. Despite this, I remember to have known that any iPhone, even the ones that are not pro versions have this in the front camera.

So, I went into research about the possibility of accessing the lidar, and it is possible to access and stream it but may cause some complications in scaling for the long run. For now, I will definitely keep that as an option, but will not continue with using that hardware for this specific project.

One of the issues that I find with my current prototype is the time it takes to create a detection. A majority of the slowness is caused by my old computer’s performance but considering that I want to setup this software to be a bigger project in the future, I already have to ensure steps to make the performance bearable. Some practical ideas that I have thought of are as follows:

1. Minimize media pipe points of data collection, specifically by removing the face entirely.

As seen above, multiple points in the face are being detected by the model, as much as this is helpful for the facial emotion detection, this slows down the model training and detection. Facial emotion is a huge part of sign language however, if I choose between inaccuracy and proper expression, the priority should be the latter. Thus, I will try to remove this from the media pipe detections.

1. User tensor flow lite.

After researching of the possible ways to optimize the detection in the tensor flow library, I have seen that it is possible to make the model friendly towards smaller devices, I will still try to find out whether there is a compromise for accuracy through efficiency.

These are the most practical things that I can change so far, I will definitely try to find more optimizations, however their successes will be defined through experimentation than anything else.