Coming from a machine learning background, I have long held the idea -throw in all variables and let regularization and cross validation fight against over-fitting.

The reason I am posting this arises from a recent study using principal component regression. Intuitively, increasing the PCR parameter (% of variance to keep) roughly amounts to removing variables. So a natural approach to take is throw in all variables and perform cross validation on the PCR parameter.

However, this approach proved sub-optimal; a later experiment showed that removing certain variables improved prediction across the board and makes prediction more stable across PCR parameter (shifting the learning curve up). This phenomenon still baffles me today.

Can anyone please comment on this, from a theoretical and applied perspective? In general, when you only care about prediction, would you consider variable selection?

  • 2
    $\begingroup$ An incredible amount of information is available on that topic on this site. Very short answer: removing variables in a way that uses supervised learning (i.e., that uses associations with $Y$) is not a very good idea. Increased accuracy is usually a mirage and will not withstand rigorous bootstrap validation. Consider sparse principal components and many other good approaches, especially penalized maximum likelihood estimation. $\endgroup$ – Frank Harrell Jun 29 '17 at 21:47
  • $\begingroup$ @FrankHarrell I guess i am more concerned with the reason why removing variables not that correlated with Y improves prediction even after thorough cross validation. $\endgroup$ – denizen of the north Jun 29 '17 at 21:52
  • 2
    $\begingroup$ @whuber You are right. Summing this up in statistical learning theory terms, including irrelevant variables increases generalization error, increasing the chance of over-fitting. Does it sound about right? $\endgroup$ – denizen of the north Jun 29 '17 at 22:27
  • 2
    $\begingroup$ Not so sure. That would be true if you knew apriori which variables are 'irrelevant'. Using associations with $Y$ to tell you which variables to exclude is akin to Maxwell's demon. It requires stealing information from the system to find out which features to use - information that might better be used for predictive accuracy. This is described in detail in my RMS course notes - see link at fharrell.com/p/blog-page.html and refer to section 4.3.1 $\endgroup$ – Frank Harrell Jun 29 '17 at 22:35
  • 1
    $\begingroup$ The image recognition example is not so straightforward. Afaik, most nns use a ridge regression penaliser (weight Decay) in addition you initialize the weights around zero and stop training when validation error increases (so this stopped training is also regularisation). Furthermore the pixels are correlated, so rather than variable selection to remove spurious pixels, they are all telling you the same thing... This is where ridge regression (or pcr) helps by avoiding large equal opposite weights. Similarly for stopped training (since you stop before weights get too big) $\endgroup$ – seanv507 Jun 29 '17 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.