I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3

I have used stepAIC in MASS package for predictors selection and I would like to see whether cross validation is a better criteria. In other words, I am trying to use cross-validation as a criterion in backward selection procedure instead of AIC.

The procedure should start with a full model, drop one predictor at a time, keep the variable if it improves CV and continue until no improvements can be made.

I think that the rfe function in the caret package may be able to do the job. I would also like to do bootstrapping if possible. If so, how should I specify it? Or are there any functions that can do that? I have tried writing my own function, but it seems to be beyond my ability. Many thanks.

  • 1
    $\begingroup$ validate in the rms package has the option of allowing backwards selection based on AIC. Do read some of the many posts here on why stepwise methods aren't so good & consider whether (a) you need to do post-hoc model selection, & (b) you ought to apply shrinkage e.g. lasso. $\endgroup$ – Scortchi - Reinstate Monica Feb 21 '14 at 12:03
  • 4
    $\begingroup$ As @Scortchi alluded, the method you proposed is not based on any good statistical principles. If you really need to do model selection (which is not obvious why this is helpful; weak variables don't hurt models), then you should combine model selection with shrinkage using a method such as elastic net. $\endgroup$ – Frank Harrell Feb 21 '14 at 12:34
  • 1
    $\begingroup$ Sorry, I misread the question (despite its being pretty clear) - you want to use a performance measure calibrated by cross-validation as a criterion in the backward selection procedure instead of AIC. Note that you'll still want to validate this procedure: see here. And that it's still stepwise, with all its faults. $\endgroup$ – Scortchi - Reinstate Monica Feb 21 '14 at 12:35
  • 2
    $\begingroup$ In addition to what Frank Harrell and Scortchi say, you should definitively read Dikran Marsupial's paper about overfitting during such massive model comparison situations: Cawley, G. C. & Talbot, N. L. C.: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Journal of Machine Learning Research, 11, 2079-2107 (2010). $\endgroup$ – cbeleites supports Monica Feb 21 '14 at 13:19
  • 1
    $\begingroup$ Thanks for the comments. As @Scortch clarified, I am trying to use cross-validation as a criterion in backward selection procedure instead of AIC. I have used backwards selection based on AIC and I would to see whether cross-validation is a better criteria. $\endgroup$ – AdrienS Feb 24 '14 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.