10.07.2025
Using FaceMesh, following the examples above and the ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model. Try to create an interaction that is surprising or one that is inspired by the project you find.
For this week’s assignment, I wanted to create a sketch that is inspired by coral reef by using falling pixels that gradual build up into a coral-like formation. At first, I planned on using Matter.js to simulate collisions and gravity effects. I imagined that the shapes would be able to bounce and stack on top of each other as it falls out of the mouth. However, after some research, I realized I needed more time figure out how to integrate Matter.js properly with p5.js and ml5.js. So, for this week’s iteration, I decided to simplify the concept and focus on capturing the visual of falling coral fragments.
I used the class example of ml5’s FaceMesh model to detect the user’s mouth position in real time. Whenever the mouth opens, pixels will fall from that region. The motion of the particles is slightly randomized in position and speed in order to achieve that organic structure of coral reef. Over time, these pixels stack unevenly on top of each other to create the rough textures of coral growth. This week I finally added sound to my sketch after receiving sound related feedback from the class for my previous assignments. I included a fairy dust sound effect that plays only when the mouth is open to enhance the whole experience. A challenge that occurred was when the sketch became laggy as more pixels compiled on the screen. I had to remove pixels from the array once they landed and loop it through the array in reverse to avoid errors.
Resources:
Class Example: https://editor.p5js.org/yining/sketches/FNjlzk9hw
Falling Pixels: https://www.youtube.com/watch?v=L4u7Zy_b868
Want to attempt: https://editor.p5js.org/piecesofuk/sketches/8cFOlol7H
[<https://brm.io/matter-js/demo/#mixed>](<https://brm.io/matter-js/demo/#mixed>)
Demo:
P5.js Sketch: https://editor.p5js.org/Janeliu13/sketches/qKeTQyM12