What are the advantages of kernel PCA over standard PCA?

Question

I want to implement an algorithm in a paper which uses kernel SVD to decompose a data matrix. So I have been reading materials about kernel methods and kernel PCA etc. But it still is very obscure to me especially when it comes to mathematical details, and I have a few questions.

1. Why kernel methods? Or, what are the benefits of kernel methods? What is the intuitive purpose?

   Is it assuming a much higher dimensional space is more realistic in real world problems and able reveal the nonlinear relations in the data, compared to non-kernel methods? According to the materials, kernel methods project the data onto a high-dimensional feature space, but they need not to compute the new feature space explicitly. Instead, it is enough to compute only the inner products between the images of all pairs of data points in the feature space. So why projecting onto a higher dimensional space?

2. On the contrary, SVD reduces the feature space. Why do they do it in different directions? Kernel methods seek higher dimension, while SVD seeks lower dimension. To me it sounds weird to combine them. According to the paper I am reading (Symeonidis et al. 2010), introducing Kernel SVD instead of SVD can address the sparsity problem in the data, improving results.

From the comparison in the figure we can see that KPCA gets an eigenvector with higher variance (eigenvalue) than PCA, I suppose? Because for the the largest difference of the projections of the points onto the eigenvector (new coordinates), KPCA is a circle and PCA is a straight line, so KPCA gets higher variance than PCA. So does it mean KPCA gets higher principal components than PCA?

Answer 1

PCA (as a dimensionality reduction technique) tries to find a low-dimensional linear subspace that the data are confined to. But it might be that the data are confined to low-dimensional nonlinear subspace. What will happen then?

Take a look at this Figure, taken from Bishop's "Pattern recognition and Machine Learning" textbook (Figure 12.16):

The data points here (on the left) are located mostly along a curve in 2D. PCA cannot reduce the dimensionality from two to one, because the points are not located along a straight line. But still, the data are "obviously" located around a one-dimensional non-linear curve. So while PCA fails, there must be another way! And indeed, kernel PCA can find this non-linear manifold and discover that the data are in fact nearly one-dimensional.

It does so by mapping the data into a higher-dimensional space. This can indeed look like a contradiction (your question #2), but it is not. The data are mapped into a higher-dimensional space, but then turn out to lie on a lower dimensional subspace of it. So you increase the dimensionality in order to be able to decrease it.

The essence of the "kernel trick" is that one does not actually need to explicitly consider the higher-dimensional space, so this potentially confusing leap in dimensionality is performed entirely undercover. The idea, however, stays the same.