Real-Time Hand Tracking in the Browser with MediaPipe

The Idea

What if you could play a puzzle game with your hands? No touchscreen, no controller — just your webcam and a gesture.

That's Snapzule. A selfie puzzle game where MediaPipe tracks your hand movements in real-time to solve puzzles.

The Tech

• MediaPipe Hands — Google's hand tracking solution, runs in the browser
• getUserMedia (MediaDevices API) — Browser camera access, low-latency
• Canvas API — For real-time rendering
• Firebase — For global leaderboard and daily challenges

How It Works

1. Camera captures video frames
2. MediaPipe processes each frame, detecting 21 hand landmarks
3. Landmark positions are mapped to game actions
4. Game state updates in real-time
5. Ghost racing lets you compete against your own best time

Challenges

1. Latency — MediaPipe processing + game rendering must happen in under 16ms for 60fps
2. Lighting — Hand detection varies wildly with lighting conditions
3. False positives — The "nothing" class is crucial for avoiding ghost inputs

Results

Live at snapzule.outshorts.in. Global leaderboard with daily challenges.