# Why is there a discrepancy between the eigenvalues of the covariance matrix (PCA) and the eigenvalues of the kernel matrix (kernel PCA)?

I've done PCA on my data matrix $$\mathbf{X}$$ which gives me i.a. the eigenvalues $$\lambda$$ and eigenvectors $$v$$ of the data covariance matrix $$C=\mathbf{X}^T \mathbf{X}$$. I'm now extending my analysis to also apply kernel PCA. Now, it can be shown that the eigenvalues of $$C$$ should be equal to the eigenvalues of the kernel matrix $$\mathbf{K}$$: $$\mathbf{K} \alpha = \lambda \alpha \\ \Leftrightarrow \mathbf{X} \mathbf{X}^T \alpha=\lambda \alpha \\ \Rightarrow \mathbf{X}^T \mathbf{X} \mathbf{X}^T \alpha=\lambda \mathbf{X}^T \alpha \\ \Leftrightarrow Cv=\lambda v$$

With $$\alpha$$ being the eigenvector of $$\mathbf{K}$$ and $$v:= \mathbf{X}^T \alpha$$ being the eigenvector of $$C$$.

After applying kernel PCA with a linear kernel (equivalent to "standard" PCA), however, the eigenvalues are not equal. I see, however, a (maybe general) relationship between $$\lambda_{PCA}$$ and $$\lambda_{KPCA}$$, because $$\overline{\lambda}_{PCA, i} = \overline{\lambda}_{KPCA, i}$$ with $$\overline{\lambda}_i = \frac{\lambda_i}{\sum_{k=1}^n \lambda_k}$$ for the $$i$$-th of the $$n$$ eigenvalues.

So why are the eigenvalues not equal? I'm using Python with sklearn.decomposition.PCA and sklearn.decomposition.KernelPCA.

• Assuming the data are centered, the covariance matrix is $\frac{1}{n} X^T X$, not $X^T X$ as you've written. This means the eigenvalues of the covariance matrix would be a constant factor of $\frac{1}{n}$ times those of the $X^T X$ (and also $X X^T)$. Does this match the discrepancy you see? – user20160 Mar 4 at 18:34
• I forgot the factor $\frac{1}{n}$ for both $C$ and $K$, but the result is the same, the eigenvalues should be equal. But I noticed, that the Gram / kernel matrix $K$ doesn't get scaled by $\frac{1}{n}$ while the covariance matrix does get scaled. This was the issue. Now, after scaling the KPCA eigenvalues there is just some minor discrepancies, which probably result from numerical issues I suppose. Thanks! – MaxGR Mar 5 at 8:51
• I realized, that the scaling of the kernel matrix eigenvalues has to be done via $\frac{1}{n-1}$. Then, the eigenvalues are exactly equal. – MaxGR Mar 5 at 9:14
• Glad it worked. If the scaling factor is $\frac{1}{n-1}$ the unbiased covariance matrix estimator is being used. – user20160 Mar 5 at 10:28

The covariance matrix $$C$$ as well as the Gram / kernel matrix $$\mathbf{K}$$ have to be scaled by $$\frac{1}{n}$$ with $$n$$ being the number of samples in the data, assuming the data is centered: $$C = \frac{1}{n} \mathbf{X}^T \mathbf{X} \\ \mathbf{K} = \frac{1}{n} \mathbf{X} \mathbf{X}^T$$ While the covariance matrix does get scaled in sklearn.decomposition.PCA (actually, PCA is computed via the SVD and the eigenvalues result from $$\lambda_i = \frac{S_i^2}{n-1}$$ with $$S$$ being the singular values), $$\mathbf{K}$$ doesn't get scaled in sklearn.decomposition.KernelPCA. After scaling the eigenvalues of $$\mathbf{K}$$ manually by $$\frac{1}{n - 1}$$, the descrepancy vanishes completely.