I'm writing the question to try to complete the circle after reading the 2 other questions on Cross Validated and the link on the third bullet point:
SVD of $X$ gives us a nice matrix factorization $USV^{T}$ that allows us to obtain the best k-rank approximation of a matrix (by zeroing the k-r smallest singular values in the matrix $S$ and obtain $X_k=US_kV^{T}$ - the subscript represents the rank of the reconstructed matrix).
This can be proved by showing that $\| X - X_k \|$ is minimized by zeroing the smallest $k$ singular values in $S$ (check third link for complete proof).
My question is the following:
- How do I go from all the above to the formulation of SVD in terms of solving the problem of residual minimization? I.e. in the first link above: $\min_{\mu, \lambda_1,\ldots, \lambda_n, \mathbf{V}_q} \sum_{i=1}^n ||x_i - \mu - \mathbf{V}_q \lambda_i||^2$
In other words why does finding the minimum of $\| X - X_k \|$ convert to solving this objective function $\min_{\mu, \lambda_1,\ldots, \lambda_n, \mathbf{V}_q} \sum_{i=1}^n ||x_i - \mu - \mathbf{V}_q \lambda_i||^2$?