How to compare PCA with KPCA for dimension reduction?

Question

Both linear principal component analysis (PCA) and kernel principal component analysis (KPCA) are unsupervised dimension reduction methods. I have a dataset with $4000$ training samples and $40000$ test samples. The dimension of their features is $200$. To fit the i.i.d assumption of PCA and KPCA, I randomly select a number of anchor samples from both training set and test set, to conduct PCA and KPCA. Suppose it is 5000 and use Gaussian kernel as the kernel function of KPCA. The $\sigma$ is selected from the range $[10:10:100,200:100,500]$.

Let their reduced dimensions $d$ be selected from the range $[10,20,...,100]$. Then based on the reduced features, I conduct a RBF-SVM classification. The hyper-parameters of RBF is selected by 5-fold cross-validation.

From the classification results, it seems to be no significant different between PCA and KPCA. But in common sense, KPCA should be better than PCA. Are there any possible explanations?

Answer 1

But in common sense, KPCA should be better than PCA.

It is very hard to state anything related to the performance of dimension reduction techniques. The main concern is that it is hard to define a metric to compare different dimension reduction techniques.

Usually, a visual examination is the best thing to do when comparing projections in low dimensional spaces (see per example What's wrong with t-SNE vs PCA for dimensional reduction using R?)