1. Introduction

<aside> 👉

If your camera preview isn’t working yet, go back and review Step 1: Camera Preview , or start directly from the https://github.com/LeeShinwon/Play-Melody/tree/feature/camera-preview branch.

</aside>

Now that you have the camera preview running, it’s time to add hand tracking using Apple’s Vision framework. Hand tracking will allow your app to detect key points such as fingertips in real time. This feature is the foundation for more advanced interactions, like recognizing pinch gestures, which we’ll build in the next step.

What you’ll learn

✅ How to enable frame capture from the camera session

✅ How to use VNDetectHumanHandPoseRequest in the Vision framework

✅ How to extract fingertip positions and convert them into screen coordinates

✅ How to pass detected hand data back to SwiftUI for visualization

By the end of this section, you’ll see red dots over your fingertips in the live camera preview.

Source Code

GitHub - LeeShinwon/Play-Melody at feature/hand-tracking