1
$\begingroup$

I've done PCA on my data matrix $ \mathbf{X} $ which gives me i.a. the eigenvalues $ \lambda $ and eigenvectors $ v $ of the data covariance matrix $ C=\mathbf{X}^T \mathbf{X} $. I'm now extending my analysis to also apply kernel PCA. Now, it can be shown that the eigenvalues of $ C $ should be equal to the eigenvalues of the kernel matrix $ \mathbf{K} $: $$ \mathbf{K} \alpha = \lambda \alpha \\ \Leftrightarrow \mathbf{X} \mathbf{X}^T \alpha=\lambda \alpha \\ \Rightarrow \mathbf{X}^T \mathbf{X} \mathbf{X}^T \alpha=\lambda \mathbf{X}^T \alpha \\ \Leftrightarrow Cv=\lambda v $$

With $ \alpha $ being the eigenvector of $ \mathbf{K} $ and $ v:= \mathbf{X}^T \alpha $ being the eigenvector of $ C $.

After applying kernel PCA with a linear kernel (equivalent to "standard" PCA), however, the eigenvalues are not equal. I see, however, a (maybe general) relationship between $ \lambda_{PCA} $ and $ \lambda_{KPCA} $, because $ \overline{\lambda}_{PCA, i} = \overline{\lambda}_{KPCA, i} $ with $ \overline{\lambda}_i = \frac{\lambda_i}{\sum_{k=1}^n \lambda_k} $ for the $ i $-th of the $ n $ eigenvalues.

So why are the eigenvalues not equal? I'm using Python with sklearn.decomposition.PCA and sklearn.decomposition.KernelPCA.

$\endgroup$
4
  • $\begingroup$ Assuming the data are centered, the covariance matrix is $\frac{1}{n} X^T X$, not $X^T X$ as you've written. This means the eigenvalues of the covariance matrix would be a constant factor of $\frac{1}{n}$ times those of the $X^T X$ (and also $X X^T)$. Does this match the discrepancy you see? $\endgroup$
    – user20160
    Mar 4, 2021 at 18:34
  • $\begingroup$ I forgot the factor $ \frac{1}{n} $ for both $ C $ and $ K $, but the result is the same, the eigenvalues should be equal. But I noticed, that the Gram / kernel matrix $ K $ doesn't get scaled by $ \frac{1}{n} $ while the covariance matrix does get scaled. This was the issue. Now, after scaling the KPCA eigenvalues there is just some minor discrepancies, which probably result from numerical issues I suppose. Thanks! $\endgroup$
    – MaxGR
    Mar 5, 2021 at 8:51
  • $\begingroup$ I realized, that the scaling of the kernel matrix eigenvalues has to be done via $ \frac{1}{n-1} $. Then, the eigenvalues are exactly equal. $\endgroup$
    – MaxGR
    Mar 5, 2021 at 9:14
  • $\begingroup$ Glad it worked. If the scaling factor is $\frac{1}{n-1}$ the unbiased covariance matrix estimator is being used. $\endgroup$
    – user20160
    Mar 5, 2021 at 10:28

1 Answer 1

1
$\begingroup$

The covariance matrix $ C $ as well as the Gram / kernel matrix $ \mathbf{K} $ have to be scaled by $ \frac{1}{n} $ with $ n $ being the number of samples in the data, assuming the data is centered: $$ C = \frac{1}{n} \mathbf{X}^T \mathbf{X} \\ \mathbf{K} = \frac{1}{n} \mathbf{X} \mathbf{X}^T $$ While the covariance matrix does get scaled in sklearn.decomposition.PCA (actually, PCA is computed via the SVD and the eigenvalues result from $ \lambda_i = \frac{S_i^2}{n-1} $ with $ S $ being the singular values), $ \mathbf{K} $ doesn't get scaled in sklearn.decomposition.KernelPCA. After scaling the eigenvalues of $ \mathbf{K} $ manually by $ \frac{1}{n - 1} $, the descrepancy vanishes completely.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.