Problem: Principal component regression for prediction. the input data has dimensionality of 500. I want to select a fixed number of components (around 10) to be used in a linear regression model.

The common practice of using PCA is to select the 'major axis' that capture most of the variance. This is done by sorting the eigenvalues and select those biggest.

Recently I read a few articles talking about those smaller eigenvalues. As opposite to the common belief that these smaller eigenvalues correspond to 'noise', they actually could play an important role in regression.

My question is : is there any established procedure to select the best eigenvectors for regression? I mean, "best" as in out-of-sample R^2 sense.

For example, my input data have 250 dimensions, I perform PCA and sort the 250 {eigenvalue eigenvector} pairs by eigenvalue. I would select the first 5 eigenvectors to capture the major variance. However, I also want to select a certain number (maybe another 5) from the rest 245 eigenvalues.

This becomes tricky because there are $c_{245}^{5}$ combinations. Some people suggested selecting the smallest 5. I am not sure if this makes sense. Intuitively the 5 new eigenvalues should be some where in the middle of the list of 245

Does anybody have any insights here?

  • $\begingroup$ Interesting question. Can you provide links to the papers or articles you are referring to? $\endgroup$ – David Kozak Nov 19 '17 at 19:15
  • 1
    $\begingroup$ This whole setup does not make any sense. $\endgroup$ – amoeba Nov 19 '17 at 19:35
  • $\begingroup$ @amoeba could you elaborate a bit why it doesn't make any sense? $\endgroup$ – user152503 Nov 19 '17 at 19:37
  • $\begingroup$ If you take the identity matrix, then "large eigenvalues" don't mean much, especially because each eigenvector will explain only a small portion of the variance. The only thing that matters is how much variance each individual component explains: so either you settle on using the first few large ones, or you need many more. $\endgroup$ – Alex R. Nov 19 '17 at 20:31
  • $\begingroup$ I don't understand what you're asking--what do you mean by the "right" eigenvectors? From your mention of PCR, it sounds like you want to use the components as input to a regression model, rather than for dimensionality reduction. Can you edit to clarify what you're trying to do? $\endgroup$ – user20160 Nov 19 '17 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.