Is Kernel PCA with linear kernel equivalent to standard PCA?

Question

If in kernel PCA I choose a linear kernel $K(\mathbf{x},\mathbf{y}) = \mathbf x^\top \mathbf y$, is the result going to be different from the ordinary linear PCA? Are the solutions fundamentally different or does some well defined relation exist?

Answer 1

Summary: kernel PCA with linear kernel is exactly equivalent to the standard PCA.

Let $\mathbf{X}$ be the centered data matrix of $N \times D$ size with $D$ variables in columns and $N$ data points in rows. Then the $D \times D$ covariance matrix is given by $\mathbf{X}^\top\mathbf{X}/(n-1)$, its eigenvectors are principal axes and eigenvalues are PC variances. At the same time, one can consider the so called Gram matrix $\mathbf{X}\mathbf{X}^\top$ of the $N \times N$ size. It is easy to see that it has the same eigenvalues (i.e. PC variances) up to the $n-1$ factor, and its eigenvectors are principal components scaled to unit norm.

This was standard PCA. Now, in kernel PCA we consider some function $\phi(x)$ that maps each data point to another vector space that usually has larger dimensionality $D_\mathrm{new}$, possibly even infinite. The idea of kernel PCA is to perform the standard PCA in this new space.

Since the dimensionality of this new space is very large (or infinite), it is hard or impossible to compute a covariance matrix. However, we can apply the second approach to PCA outlined above. Indeed, Gram matrix will still be of the same manageable $N \times N$ size. Elements of this matrix are given by $\phi(\mathbf{x}_i)\phi(\mathbf{x}_j)$, which we will call kernel function $K(\mathbf{x}_i,\mathbf{x}_j)=\phi(\mathbf{x}_i)\phi(\mathbf{x}_j)$. This is what is known as the kernel trick: one actually does not ever need to compute $\phi()$, but only $K()$. Eigenvectors of this Gram matrix will be the principal components in the target space, the ones we are interested in.

The answer to your question now becomes obvious. If $K(x,y)=\mathbf{x}^\top \mathbf{y}$, then the kernel Gram matrix reduces to $\mathbf{X} \mathbf{X}^\top$ which is equal to the standard Gram matrix, and hence the principal components will not change.

A very readable reference is Scholkopf B, Smola A, and Müller KR, Kernel principal component analysis, 1999, and note that e.g. in Figure 1 they explicitly refer to standard PCA as the one using dot product as a kernel function:

Answer 2

In addition to amoeba's nice answer, there is an even simpler way to see the equivalence. Again let $X$ be the data matrix of $N \times D$ size with $D$ variables in columns and $N$ data points in rows. Standard PCA corresponds to taking a singular value decomposition of the matrix $X = U \Sigma V^\top$ with $U$ the principal components of $X$. The singular value decomposition of the linear kernel $XX^\top = U \Sigma^2 U^\top$ has the same left singular vectors and so the same principal components.

Answer 3

It seems to me that that a KPCA with linear kernel should be the same as the simple PCA.

The covariance matrix that you are going to get the eigenvalues from is the same:

$$ linearKPCA_{matrix} = \frac{1}{l} \sum_{j=1}^{l}K(x_{j},x_{j}) = \frac{1}{l} \sum_{j=1}^{l}x_{j}x_{j}^T = PCA_{matrix} $$

You can check with more details here.