Problem
When a deforming object is captured by a moving camera, non-rigid structure from motion (NRSFM) aims to obtain the varying 3D structure and camera motion from uncalibrated 2D point tracks.
However, if the object deforms arbitrarily, reconstruction becomes under-constrained. Previous methods assume that the deforming structure could be reconstructed by a linear basis, the assumption Which is invalid in most real cases.
In this thesis, we aims to focus NRSFM on a specific defoming object, for example, human faces in the experiments, so that a pre-trained prior model could be available.
We categorize 2D point tracks into two groups, namely, well-defined landmarks and detected interesting points, and tackle the problem by two steps. First, we model the 3D structure of well-defined landmarks in probabilistic framework
and then optimize it by MAP manner to obtain initial structure and camera motion. Second, we jointly optimize the 3D structure of well-defined landmarks and detected interesting points and refine camera motion by
introducing suface constraint and distance constraint into final optimization. We further reconstruct a complete surface by fitting the surface model to the estimated structer points and a novel view of surface could be synthesized.
Figure 1. The overview of our method. We extract two kind of feature points at first, namely, well-defined landmarks and detected interesting points, and then forward them to our two-step NRSFM algorithm.
Furthermore, we recover the complete surface by constraining the surface to pass the reconstructed structure points.
Result
We evaluate our method on BU-3DFE and BU-4DFE datasets, both of which contain images and associated 3d models for several individuals with six expressions.
Since the models in BU-4DFE are temporally varying 3D human faces with expression deformations, we generate testing videos from BU-4DFE and use BU-3dFE for training.
Figure 2. Estimated 3D structures of well-defined points by our MAP formulation. The leftmost column contains input images and 2D track points.
The two columns in the middle are the 3D structures projected in different views. The back-projection errors are shown at the rightmost column.
The blue crosses are 2D track points, and the red circles are beck-projections of 3D structre.
Figure 3. Estimated 3D structures after our final optimization step. The leftmost column contains input images and 2D track points.
The two columns in the middle are the 3D structures projected in different views. The back-projection errors are shown at the rightmost column.
The blue crosses are 2D track points, and the red circles are are beck-projections of 3D structre.
Video of reconstructed surface
Master's Thesis
Tsung-Yu Lin, and Shang-Hong Lai, Combining Shape Prior with Non-Rigid SfM for Recovering Specific 3D Deformable Surface, Master's Thesis
[pdf]