I have learnt PCA in my data mining course and I know it is used for dimensionality reduction but I am confused about it how does it used for making 3D Morphable Model.
Here is the paper link: Volker Blanz and Thomas Vetter, 1999, A Morphable Model For The Synthesis Of 3D Faces.
I have read this paper but I did not get the concept of how PCA works in 3DMM. Can anybody explain it in layman's terms?
3DMM modelling pipeline is quite complex. Here is a quick summary of how it works.
First, you have to train the model on a set of 3D scans of faces. They are the registered 3D point clouds, i.e. all clouds contain equal numbers of points $N$, and the points correspond to semantically similar parts, e.g. the point #121 always corresponds to left eye outer corner. This is quite hard to achieve, and the optical flow algorithm in 3D+colour space is employed to parameterize scans (see Section 5). Thus, each point has the corresponding colour. (This means that relative calibration of colour and 3D sensors should be known during capture).
Then, PCA is applied to a set of faces on the $6D \times N$ features (3 coordinates and RGB colour for each of the $N$ points). It finds a subspace of feasible faces (Section 3). As a result, each 3D face can be transformed to a vector of latent variables (PCA space) by taking projections on eigenvectors (i.e. $(6D \times N)$-dimensional basis vectors that correspond to some hallucinated faces). Inversely, given a latent description of a face (any vector in PCA space), the corresponding 3D model can be re-generated. Since the latent space dimensionality is lower, compressing-decompressing will incur a reconstruction error.
At test time, however, we do not have 3D shapes, so cannot compute a latent representation directly. Thus, we optimize over all vectors in the PCA space and all possible camera poses, so that the rendered image would match the input image. The paper quantifies this matching as just the Euclidean distance between images (Section 4). There's more than that, though. It is important to account for prior probabilities of latent vectors (thus, respectively, face models). Since PCA essentially fits the normal distribution, the distribution naturally follows from its computation, e.g. for each eigenvalue, the most probable value is around zero.
tl;dr: PCA is applied to estimate the prior distribution in the space of coloured 3D scans; then at test time you take a plausible sample that can be rendered similarly to the input image.
