Why is there a reconstruction loss in PCA with orthonormal eigenvectors?

Question

I've already read How to reverse PCA and reconstruct original variables from several principal components? and I understand conceptually and visually why there has to be a reconstruction loss.

However, if we have a data matrix $\mathbf X$ and its orthonormal eigenvectors $\mathbf {V}$ and then take the first $k$ eigenvectors and make a low-rank approximation: $\mathbf Z=\mathbf {XV_k}$

With $\mathbf {V_k}$ being orthonormal, shouldn't it be $\mathbf X=\mathbf {XV_k V^{T}_{k}}$ because $I = \mathbf {V_k V^{T}_{k}}$?

Maybe someone can provide an example why exactly this is wrong? A perfect answer would just provide a simple numerical example and/or an explanation of what I'm missing here.

Answer 1

Your confusion arises from the fact that we don't have $V_kV_k^T\neq I$ in general. If $k=K$, i.e. number of features, this is guaranteed to be true. Orthogonality of eigenvectors produces $V_k^T V_k=I$, not $V_kV_k^T= I$. Because, $$V_k^T V_k=\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}=\begin{bmatrix}v_1^Tv_1&v_1^Tv_2&\cdots&v_1^Tv_k\\v_2^Tv_1&v_2^Tv_2&\cdots&v_2^Tv_k\\\vdots&&\ddots&\vdots\\v_k^Tv_1&v_k^Tv_2&\cdots&v_k^Tv_k\end{bmatrix}=\begin{bmatrix}1&0&\cdots&0\\0&1&\cdots&0\\\vdots&&\ddots&\vdots\\0&0&\cdots&1\end{bmatrix}=I$$ But, in $V_kV_k^T$, we don't have the above scenario: $$V_kV_k^T=\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}=v_1v_1^T+\cdots+v_kv_k^T\underbrace{\neq}_{\text{in general}} I$$ which is composed of outer products of eigenvectors; not inner products as in previous case.

Answer 2

To give a short answer, If you take only k eigenvectors then $V^{k}$ is a $nxk$ Matrix. Therefore multiplying by $V^{kT}$ (put on the right side, i.e. $V^{k}V^{kT}$) produces a $nxn$ Matrix which is not the $nxn$ identity matrix I. This is because when you take only a subset of eigenvectors then the reduced eigenvector matrix is not rank n (it is rank k, so full column rank) and as such is not invertibile (more precisely here we must say “it does not have any right inverse”). Indeed $V^{kT}$ is just the pseudoinverse of $V^{k}$ (Moore Penrose pseudoinverse), which is (if you wish, in a few words) the “best proxy of an inverse” but not an inverse. Indeed in this case, only $V^{kT}V^{k}$ is a $kxk$ Identity matrix, as $V^{kT}$ is the left inverse of $V^{k}$, but multiplying $V^{k}$ by its left inverse put on the right side produces a $nxn$ Matrix that is different from I (because, as said, it is the left inverse not the right inverse). For more references, check Wikipedia Moore Penrose pseudoinverse and generalized inverse and very good source for inverse of a matrix including right and left inverse