For an observations matrix $X$, I use PCA to reduce the dimension of the data to $L$. I know that in this case, PCA is guaranteed to minimise the mean reconstruction error. In Wikipedia notation, something like $$\|X-T_LW_L^T\|^2_2$$ where $W$ are the loadings, $T$ are the scores, and the subscript $L$ indicates that only $L$ components are used.

I can also ask how much of the total variance is explained by these $L$ components.

My question is, is there a mathematical relation between the two? Assuming I have only the squared error and the covariance matrix for $X$, can I compute the explained variance?


If your $\newcommand{\X}{\mathbf X}\X$ is a $n\times d$ matrix with column means subtracted and $\newcommand{\W}{\mathbf W}\W_L$ is a $d \times L$ matrix consisting of $L$ principal directions (eigenvectors of the covariance matrix), then reconstruction error is given by $$\mathrm{Error}^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2.$$ Note that this is not a mean squared error, it is the sum of squared errors. Here $\X\W_L$ is what you called $\mathbf T_L$ in your question and $\|\cdot\|$ denotes Frobenius norm.

The proportion of explained variance in PCA can be defined as $$\text{Proportion of explained variance} = \frac{\|\X\W_L\|^2}{\|\X\|^2},$$ i.e. it is a ratio of scores' sum of squares to the overall sum of squares (or equivalently, a ratio of scores' total variance to the overall total variance).

To see the connection between them, we need the following identity: $$\|\X\|^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2 + \|\X\W_L^\vphantom{\top}\W_L^\top\|^2 = \|\X - \X\W_L^\vphantom{\top}\W_L^\top\|^2 + \|\X\W_L\|^2.$$ It might look mysterious, but is actually nothing else than Pythagoras theorem; see an informal explanation in my answer to Making sense of principal component analysis, eigenvectors & eigenvalues and a formal explanation in my answer to PCA objective function: what is the connection between maximizing variance and minimizing error?

We now see that $$\text{Proportion of explained variance} = 1-\frac{ \text{Error}^2}{\|\X\|^2}.$$

If you know the covariance matrix $\mathbf C = \frac{1}{n-1}\X^\top \X$, then you can compute the total variance $\operatorname{tr}\mathbf C$. The squared norm $\|\X\|^2$ is then given by the total variance multiplied by $n-1$. So we finally obtain

$$\text{Proportion of explained variance} = 1-\frac{\text{Error}^2}{(n-1)\operatorname{tr}\mathbf C}.$$

  • 1
    $\begingroup$ Great answer! But I believe it should be 1 - (Error^2)/||X||^2 (and same for the last equation). $\endgroup$
    – Hypercube
    Nov 26 '19 at 3:04
  • $\begingroup$ @Hypercube Thanks, you are of course right! Edited. $\endgroup$
    – amoeba
    Nov 26 '19 at 6:49

Let $ W= \left[ \begin{matrix}W_1 & W_2\end{matrix} \right]$ and $T= \left[ \begin{matrix}T_1 & T_2\end{matrix} \right]$ be the partitioned $W\in\mathbb{R}^{M\times M}$ and $T\in\mathbb{R}^{N\times M}$ matrices, where $W_1$ and $T_1$ are the submatrices containing columns $1\dots L$ of $W$ and $T$ respectively. The data matrix can be written in terms of these submatrices as $$ X = TW^\top = T_1 W_1^\top + T_2 W_2^\top $$ which implies $T_1=XW_1$ and $T_2=XW_2$. On the other hand the reconstructed data matrix is defined to be $\hat X=T_1W_1^\top$. Using all these equations we get that the error matrix $E=X-\hat X$ is $$ E = X - \hat X = X - T_1 W_1^\top = T_2 W_2^\top = XW_2 W_2^\top $$ The rows of $E$ are the error vectors, that is the difference between the original data vectors—the rows of $X$—and the reconstructed vectors—the rows of $\hat X$. Therefore the trace of $EE^\top$, which is the sum of the squared lengths of the error vectors, is the total squared reconstruction error.

So what is the trace of $EE^\top$?

We know that $$ E^\top E = W_2 W_2^\top X^\top XW_2 W_2^\top $$ and using $(N-1)S=X^\top X$, where $S$ is the covariance matrix of $X$, in the above equation $$ \begin{align} E^\top E &= (N-1)W_2 W_2^\top SW_2 W_2^\top \\ &= (N-1)W_2 W_2^\top W_2 D_2 W_2^\top \\ &= (N-1)W_2 D_2 W_2^\top \end{align} $$ where $D_2$ is the diagonal matrix with the lowest $(M-L)$ eigenvalues of $S$. Finally, using the properties of the trace we get $$ \begin{align} \text{trace}(EE^\top) &= \text{trace}(E^\top E) \\ &= \text{trace}((N-1)W_2 D_2 W_2^\top) \\ &= \text{trace}((N-1)W_2^\top W_2 D_2) \\ &= \text{trace}((N-1) D_2) \end{align} $$ This is telling us that total squared reconstruction error is just the sum of the lowest $(M-L)$ eigenvalues of $S$ multiplied by the constant $(N-1)$. Therefore minimising the total squared reconstruction error is equivalent to maximising the sum of the $L$ highest eigenvalues of $S$, which in turn is equivalent to maximising the variance of the first $L$ principal component scores.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.