0
$\begingroup$

Both linear principal component analysis (PCA) and kernel principal component analysis (KPCA) are unsupervised dimension reduction methods. I have a dataset with $4000$ training samples and $40000$ test samples. The dimension of their features is $200$. To fit the i.i.d assumption of PCA and KPCA, I randomly select a number of anchor samples from both training set and test set, to conduct PCA and KPCA. Suppose it is 5000 and use Gaussian kernel as the kernel function of KPCA. The $\sigma$ is selected from the range $[10:10:100,200:100,500]$.

Let their reduced dimensions $d$ be selected from the range $[10,20,...,100]$. Then based on the reduced features, I conduct a RBF-SVM classification. The hyper-parameters of RBF is selected by 5-fold cross-validation.

From the classification results, it seems to be no significant different between PCA and KPCA. But in common sense, KPCA should be better than PCA. Are there any possible explanations?

$\endgroup$
  • $\begingroup$ I didn't quite understand the title. What is dimension dimension? $\endgroup$ – ttnphns Jan 27 '16 at 9:22
  • $\begingroup$ I guess it should read dimension reduction? $\endgroup$ – Vincent Guillemot Jan 27 '16 at 9:27
  • $\begingroup$ Did you also compare the classification results with the situation where there is no dimension reduction? $\endgroup$ – Vincent Guillemot Jan 27 '16 at 9:37
  • $\begingroup$ @ttnphns, I have revised it. Please kindly check it. Thanks. $\endgroup$ – mining Jan 27 '16 at 9:56
  • $\begingroup$ @VincentGuillemot, thanks! The result without dimension reduction seems to be a little better than that with dimension reduction, say about 0.1%. $\endgroup$ – mining Jan 27 '16 at 9:57
1
$\begingroup$

But in common sense, KPCA should be better than PCA.

It is very hard to state anything related to the performance of dimension reduction techniques. The main concern is that it is hard to define a metric to compare different dimension reduction techniques.

Usually, a visual examination is the best thing to do when comparing projections in low dimensional spaces (see per example What's wrong with t-SNE vs PCA for dimensional reduction using R?)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer! Please check this link [quora.com/…, there are some discussions and comments! $\endgroup$ – mining Aug 4 '16 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.