Ruihang Zhang, Felix Taubner, Pooja Ravi, Kiriakos N. Kutulakos, David B. Lindell
A new method for 6-DoF pose tracking from monocular video using a video diffusion model, requiring only a single marked pixel in the first frame.
Tracking the 6-DoF pose of objects and surfaces from monocular video is a long-standing computer vision problem. Existing methods require additional inputs like 3D models, depth maps, or object masks, and struggle with challenging materials like textureless, transparent, or reflective surfaces.
ProxyPose recasts 6-DoF pose tracking as video-to-video translation. Given only a video and a single marked pixel, a fine-tuned video diffusion model translates the input into a proxy video—a synthetic video depicting a colored polyhedron undergoing the same local rigid-body motion as the surface region at the marked pixel. Since the proxy's geometry and appearance are known, recovering its full 6-DoF trajectory reduces to classical pose estimation with off-the-shelf solvers.
ProxyPose achieves state-of-the-art 6-DoF pose tracking accuracy without the additional inputs required by competing methods, after fine-tuning the video model only on synthetic data. It further demonstrates extensions to face tracking, camera pose estimation, and challenging in-the-wild scenes beyond the reach of existing approaches.