1
$\begingroup$

I've already read How to reverse PCA and reconstruct original variables from several principal components? and I understand conceptually and visually why there has to be a reconstruction loss.

However, if we have a data matrix $\mathbf X$ and its orthonormal eigenvectors $\mathbf {V}$ and then take the first $k$ eigenvectors and make a low-rank approximation: $\mathbf Z=\mathbf {XV_k}$

With $\mathbf {V_k}$ being orthonormal, shouldn't it be $\mathbf X=\mathbf {XV_k V^{T}_{k}}$ because $I = \mathbf {V_k V^{T}_{k}}$?

Maybe someone can provide an example why exactly this is wrong? A perfect answer would just provide a simple numerical example and/or an explanation of what I'm missing here.

$\endgroup$
3
$\begingroup$

Your confusion arises from the fact that we don't have $V_kV_k^T\neq I$ in general. If $k=K$, i.e. number of features, this is guaranteed to be true. Orthogonality of eigenvectors produces $V_k^T V_k=I$, not $V_kV_k^T= I$. Because, $$V_k^T V_k=\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}=\begin{bmatrix}v_1^Tv_1&v_1^Tv_2&\cdots&v_1^Tv_k\\v_2^Tv_1&v_2^Tv_2&\cdots&v_2^Tv_k\\\vdots&&\ddots&\vdots\\v_k^Tv_1&v_k^Tv_2&\cdots&v_k^Tv_k\end{bmatrix}=\begin{bmatrix}1&0&\cdots&0\\0&1&\cdots&0\\\vdots&&\ddots&\vdots\\0&0&\cdots&1\end{bmatrix}=I$$ But, in $V_kV_k^T$, we don't have the above scenario: $$V_kV_k^T=\begin{bmatrix}v_1&\cdots&v_k\end{bmatrix}\begin{bmatrix}v_1^T\\\vdots \\v_k^T\end{bmatrix}=v_1v_1^T+\cdots+v_kv_k^T\underbrace{\neq}_{\text{in general}} I$$ which is composed of outer products of eigenvectors; not inner products as in previous case.

$\endgroup$
3
$\begingroup$

To give a short answer, If you take only k eigenvectors then $V^{k}$ is a $nxk$ Matrix. Therefore multiplying by $V^{kT}$ (put on the right side, i.e. $V^{k}V^{kT}$) produces a $nxn$ Matrix which is not the $nxn$ identity matrix I. This is because when you take only a subset of eigenvectors then the reduced eigenvector matrix is not rank n (it is rank k, so full column rank) and as such is not invertibile (more precisely here we must say “it does not have any right inverse”). Indeed $V^{kT}$ is just the pseudoinverse of $V^{k}$ (Moore Penrose pseudoinverse), which is (if you wish, in a few words) the “best proxy of an inverse” but not an inverse. Indeed in this case, only $V^{kT}V^{k}$ is a $kxk$ Identity matrix, as $V^{kT}$ is the left inverse of $V^{k}$, but multiplying $V^{k}$ by its left inverse put on the right side produces a $nxn$ Matrix that is different from I (because, as said, it is the left inverse not the right inverse). For more references, check Wikipedia Moore Penrose pseudoinverse and generalized inverse and very good source for inverse of a matrix including right and left inverse

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.