TouchDesigner with MediaPipe and StreamDiffusion plugins
An interactive system where users “paint” on a canvas with their fingertips, and an AI generates imagery along the gesture path in real time.
This project explores real-time AI image generation through simple hand-gesture interaction. Using MediaPipe for hand tracking and StreamDiffusion for fast image synthesis, the system allows users to “paint” in the air and instantly generate visuals in the style of Vincent van Gogh. Instead of drawing with traditional tools, users move their hands to create colorful strokes, and the AI transforms these movements into textures inspired by Van Gogh’s brushwork, such as swirling skies, bold outlines, or expressive color contrasts.
The interface is intentionally minimal: users draw directly in the air with the tip of one hand, switch between different Van Gogh reference images by raising the other hand in a “point up” gesture, and hold a fist to briefly freeze the output and capture a static frame. Each reference painting acts as an IP adapter input, guiding the generated results toward specific visual characteristics—Whether it’s the yellows and blues from Starry Night, the floral structure from Irises, or the delicate blossoms from Almond Blossom.
Rather than aiming for a polished final product, this project focuses on experimentation. It looks at how AI tools can be used in a live, playful, and responsive way. The goal is not to replicate Van Gogh’s style perfectly, but to explore how his visual language can be reinterpreted through movement, immediacy, and user interaction. The result is a small interactive system that lets people improvise with AI-assisted painting in an intuitive and accessible manner.
I’m quite interested in AI-generated imagery, especially when it can respond in real time to user interactions. I think it’s really exciting to see visuals or video change instantly based on gestures or movements, and it seems like a perfect fit for live performance contexts. While browsing online, I also came across a few similar interactive AI-art projects, and I realized this is a field I haven’t explored much yet. That made me want to experiment without setting expectations for the final outcome.
So instead of setting any specific goal for the outcome, I simply see this project as a chance to explore: to learn how tools like MediaPipe and StreamDiffusion work together, to see what kinds of images AI can generate on the fly, and to experience the process of turning bare hand movements into something visual, painterly, and alive.
Process
Step 1: Hand Detection & Gesture Extraction