Kernel PCA increases dimensionality compared with PCA?

Question

I was trying to use sklearn to perform kernel PCA with 28*28 = 784 dims data. At first I used PCA to reduce dimensionality and I chose to reduce to k dimensions where k could explain 95% of the variance. PCA gave me k = 174. Later on I tried kernel PCA with polynomial kernel of degree 3 and similarly, using the explained-variance approach I got k = 1993. Since 1993 > 784, kernel PCA actually increased the dimensionality, which was against my intention.

Also, I used 5000 data for training and Kernel PCA gives me 5000 eigenvectors, from which I selected k = 1993. Why did kernel PCA give me 5000 eigenvectors? Why Kernel PCA increased dimensionality compared to PCA?

Answer 1

Why Kernel PCA increased dimensionality compared to PCA?

That's the nature of the kernels: they take input in one dimension and translate it to other dimension. In your case, it's easy to see applying the polynomial kernel increases the dimensionality of the problem.

Now, I'm not sure what's the implementation you're using for kernel PCA, but 95% of explained variance in a higher dimensional space may, indeed, require that many features. Have you tried to test the procedure yourself?

library(kernlab)

#data = as.matrix(iris[,-5]) #input your data
kpcl = kpca(data, kernel = "vanilladot", kpar = list())
kpcp = kpca(data, kernel = "polydot", kpar = list(degree = 3))

#this gives the explained variance
cumsum(eig(kpcl))/sum(eig(kpcl)) 
cumsum(eig(kpcp))/sum(eig(kpcp))

Why did kernel PCA give me 5000 eigenvectors?

If you are using kernlab::kpca, the function pcv returns the principal component vectors arranged columnwise. Ordinary PCA would return 5000 rows aswell, and I guess the same should be expected in other implementations. Check this:

#both are n_samples by n_components matrices
pcv(kpcl) 
pcv(kpcp)