# Why is there a reconstruction loss in PCA with orthonormal eigenvectors?

I've already read How to reverse PCA and reconstruct original variables from several principal components? and I understand conceptually and visually why there has to be a reconstruction loss.

However, if we have a data matrix $$\mathbf X$$ and its orthonormal eigenvectors $$\mathbf {V}$$ and then take the first $$k$$ eigenvectors and make a low-rank approximation: $$\mathbf Z=\mathbf {XV_k}$$

With $$\mathbf {V_k}$$ being orthonormal, shouldn't it be $$\mathbf X=\mathbf {XV_k V^{T}_{k}}$$ because $$I = \mathbf {V_k V^{T}_{k}}$$?

Maybe someone can provide an example why exactly this is wrong? A perfect answer would just provide a simple numerical example and/or an explanation of what I'm missing here.

Your confusion arises from the fact that we don't have $$V_kV_k^T\neq I$$ in general. If $$k=K$$, i.e. number of features, this is guaranteed to be true. Orthogonality of eigenvectors produces $$V_k^T V_k=I$$, not $$V_kV_k^T= I$$. Because, $$V_k^T V_k=\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}=\begin{bmatrix}v_1^Tv_1&v_1^Tv_2&\cdots&v_1^Tv_k\\v_2^Tv_1&v_2^Tv_2&\cdots&v_2^Tv_k\\\vdots&&\ddots&\vdots\\v_k^Tv_1&v_k^Tv_2&\cdots&v_k^Tv_k\end{bmatrix}=\begin{bmatrix}1&0&\cdots&0\\0&1&\cdots&0\\\vdots&&\ddots&\vdots\\0&0&\cdots&1\end{bmatrix}=I$$ But, in $$V_kV_k^T$$, we don't have the above scenario: $$V_kV_k^T=\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}=v_1v_1^T+\cdots+v_kv_k^T\underbrace{\neq}_{\text{in general}} I$$ which is composed of outer products of eigenvectors; not inner products as in previous case.
To give a short answer, If you take only k eigenvectors then $$V^{k}$$ is a $$nxk$$ Matrix. Therefore multiplying by $$V^{kT}$$ (put on the right side, i.e. $$V^{k}V^{kT}$$) produces a $$nxn$$ Matrix which is not the $$nxn$$ identity matrix I. This is because when you take only a subset of eigenvectors then the reduced eigenvector matrix is not rank n (it is rank k, so full column rank) and as such is not invertibile (more precisely here we must say “it does not have any right inverse”). Indeed $$V^{kT}$$ is just the pseudoinverse of $$V^{k}$$ (Moore Penrose pseudoinverse), which is (if you wish, in a few words) the “best proxy of an inverse” but not an inverse. Indeed in this case, only $$V^{kT}V^{k}$$ is a $$kxk$$ Identity matrix, as $$V^{kT}$$ is the left inverse of $$V^{k}$$, but multiplying $$V^{k}$$ by its left inverse put on the right side produces a $$nxn$$ Matrix that is different from I (because, as said, it is the left inverse not the right inverse). For more references, check Wikipedia Moore Penrose pseudoinverse and generalized inverse and very good source for inverse of a matrix including right and left inverse