code: https://editor.p5js.org/Janeliu13/sketches/nV3RztLow
Demo:
For this week, I wanted to combine the neural network system with the concept from my previous bubble interaction. I wanted to train the model to recognize three different hand gestures and according to the shape of the hand gesture, bubbles in the corresponding shape will appear. The shapes I trained it on was triangle, circle, and square. At first, choosing what hand pose to train it on was hard because sometimes the neural network could not differentiate them. So after experimenting with a few options, I landed with these three:
An issue I came across was that my model's accuracy just wasn't very good. Tt would frequently misclassify gestures or seem confused between similar hand positions. At first, I had set the training epochs to 20, thinking that would be sufficient, but the model wasn't learning the patterns well enough. I realized I might not be giving the model enough time to learn from the data so I increased the epochs to 50. This was much more reliable and consistent. Everything else went pretty smoothly. I took the bubble particles effect I created for week 3 and made changes such as the shape, color, opacity, and speed. Something that I would want to change about this iteration is to be able to save the trained data and not have to run it again every single time. But I guess for the sake of this weeks assignment, it is better to have to train it every time in order to show the process. Another improvement would be to have a automatic recording button so it will capture the training data without me having to press the button several times (I was inspired after seeing Jiawen’s sketch). This would be much more convenient for me since mine requires both hands to be making the hand gestures. In my demo video, I had to get someone to help me press on the training buttons while I had the hand gestures.
Resources:
Previous sketch: https://editor.p5js.org/Janeliu13/sketches/9BNox9DEm
Class example sketch: https://editor.p5js.org/ml5/sketches/rR51vvi-u