How to select a number of components to retain in kernel PCA?

Question

I'm using kpca function from kernlab and try to get the proportion of variance explained by each component as in standard PCA. I don't select the number of features a priori since I would like to check their contribution. However, I get 124 components which is much more than my original dataset which has 10 covariates. On the other hand, there are no large eigenvalues that would allow me to cut at some level. Is there any alternative to select a number of features?

Here are my eigenvalues:

     0.040155876 0.031142483 0.029499281 0.024417995 0.023562053 0.021718055
     0.020310953 0.018984183 0.017789973 0.017311123 0.015484136 0.015346860 
     0.015005007 0.013791102 0.013291260 0.012670090 0.012180261 0.011882593 
     0.011523336 0.011107896 0.011045174 0.010477924 0.010251907 0.009882142 
     0.009606943 0.009529857 0.009362611 0.009340580 0.009062668 0.008987593 
     0.008699146 0.008670243 0.008549814 0.008398879 0.008214842 0.008091366 
     0.008029260 0.007924718 0.007857977 0.007771030 0.007691762 0.007657295 
     0.007582320 0.007510590 0.007470620 0.007404477 0.007285695 0.007246443 
     0.007134445 0.007087406 0.006956178 0.006935525 0.006898103 0.006864934 
     0.006653101 0.006605607 0.006585557 0.006513107 0.006395417 0.006207376 
     0.006171564 0.006043939 0.006023738 0.005955121 0.005894706 0.005788706 
     0.005714630 0.005700367 0.005601950 0.005550044 0.005441031 0.005410300 
     0.005367971 0.005246899 0.005161450 0.005093251 0.005026677 0.004984414 
     0.004866770 0.004674961 0.004655324 0.004644769 0.004529852 0.004447371 
     0.004411176 0.004338879 0.004258299 0.004135511 0.004078752 0.003985330 
     0.003902527 0.003838939 0.003734150 0.003582305 0.003547204 0.003485095 
     0.003440328 0.003397815 0.003363152 0.003246147 0.003223031 0.003184239
     0.003091351 0.002938476 0.002868938 0.002765338 0.002645138 0.002572225 
     0.002544704 0.002466896 0.002419687 0.002298704 0.002187789 0.002089151 
     0.002019031 0.001957721 0.001908535 0.001887064 0.001760442 0.001705021
     0.001587056 0.001536336 0.001228544 0.001079629

Answer 1

The reason you get 124 components even though you only had 10 original features is (probably) because you have 124 samples. In kernel PCA, the data are mapped to a space which is very high dimensional (has many more than 10 dimensions), and so the number of PCs is only limited by the number of samples.

Now, your eigenvalues are actually not so uniform as you seem to think. Here is the plot of your data:

One could argue that there is some sort of an "elbow" around 15 components, and that after around 20 components the spectrum becomes very monotonic. So 15-20 components seems like a reasonable number on the basis of eigenvalues only.

However, if you want to use kPCA as a first step of some classification or decoding algorithms, then it is always a better idea to select the number of components to retain via cross-validation.

Answer 2

Here's the explained variance plot $\frac{\sum_{i=1}^k\lambda_i}{\sum_{i=1}^{124}\lambda_i}$.

You need 90 PCs to explain 90% of the variance. In my opinion, your kernel is not so good. Maybe you should try other kernels and see if this plot become more like in the picture below, which is from this paper: Williams, Christopher KI. "On a connection between kernel PCA and metric multidimensional scaling." Machine Learning 46.1-3 (2002): 11-19.

It's good when the explained variance is very steep on the left.