My project is title Bridge, it is an AI powered sign language interpreter, and this week I have created monumental steps in the creation of my software. I have also done some user testing which gave me a lot of insights in the use of the software.

To pick up where I left off from the last week, I was having trouble making my trained model accurate enough, luckily the methods that I set from the last week helped me push through with the accuracy of my model.

1. I instantly tried to remove the padding that I have done to the raw data
   1. Although I have done backups of my data, it is saved in a different drive so I couldn’t access it easily.
   2. Therefore, I edited the sorting process of the data instead, instead of including the datapoints from 1-50 (including padded data), I changed it to 30 at first and then 35 as I decided that it was a little too short.
2. After researching some more about LSTM layers, I tied different experiments with them
   1. I increased the number of LSTM layers only
   2. I increased the number of Dense layer only
   3. I increased the number of both
   4. What I decided on was adding two extra LSTM layers with the same sizes and one extra dense layer with a size of 64
3. I also tried to do some experiments with the different hyperparameters
   1. In my previous prototypes, I have decided that I the Adam optimizer is the way to go so I did not move that
   2. I changed the epochs from 2000 to 200 to prevent overfitting, I observed that it was already overfitting the moment it reaches over 200 epochs
   3. I decreased the batch sizes as I prioritize specificity vs generalizability

From all of these adjustments, I have made it possible to shift my accuracy from the 2% to 90%

Since the presentation is going closer at that time, I decided to work on the UI.

I already had a template using pygame to display the video using the pygame built-in camera. however, I seem to see that the 90% accuracy was not being reflected in real time.

After some time of debugging, I realized that the rate of the predictions were happening much faster than I was finishing the gesture and thus was feeding the model the wrong information. Comparing it to the rate when I was recording the data myself, I knew it was too fast. So, I tried to slow down the process of capturing the frames. I went through 2 routes:

1. I tried to create a separate thread that collated the data at a specific time with the time library,
   1. however, it simply just collected all datapoints at a slow rate instead of using the present frame at the time a tick happens.
2. I went back to the same framework that I was using for data collection which was through the use of OpenCV.
   1. however the camera recording functionality was not affected by limiting the framerate.
   2. what made the lag in the data collection was the numpy saving process, so it was separate.

I thought of a solution where I just simulated a lag of data collection. So, I made a counter that took data only after a few frames had passed. This worked really well as it was now adjusted to the rate of saving the frames, thus simulating the situation of the data collection

Next, I had to create the UI. As I said earlier, I already had a template, however it lacked the buttons and the text and the promised speech to text.