I have seen the following commonly used: 1. fit a model with all variables, 2. in a single reduction step, remove from the model all variables at once that do not fit some criteria (p-value, whatever), 3. calibrate the reduced model, in my case to a new data set, 4. check model results and hopefully everything went well and you can stop.

I'm betting this will perform better than stepwise, especially when used with data splitting, but would appreciate a name for this procedure if it exists and perhaps a reference related to it so that I can learn more. Maybe it is so simple/bad/obvious that no one has bothered? -WVG

  • $\begingroup$ Two correlated predictors will tend to have high p-values (neither explains much variation in the response given the presence of the other in the model), so you could end up removing both when stepwise would have kept one in, thus degrading predictive performance severely. $\endgroup$ – Scortchi - Reinstate Monica Oct 25 '13 at 12:28
  • $\begingroup$ Thanks for the input. I have examined multicollinearity among predictors and it is very low for all variables. In my case, only 2 independent variables were eliminated, and they were not expected to be correlated (very different). I would rather eliminate them together in one step, but I would like to know a bit more. $\endgroup$ – wvguy8258 Oct 25 '13 at 12:32
  • 1
    $\begingroup$ If all predictors were orthogonal then the only effect of removing some would be to bias the estimate of the error variance, making the calculated p-values & confidence intervals misleading (as with stepwise); which hardly seems an advantage. There's plenty under the model selection tag (e.g. this & this) about the perils of model reduction & suggestions for methods that aren't terrible. $\endgroup$ – Scortchi - Reinstate Monica Oct 25 '13 at 12:50
  • $\begingroup$ Does anyone know the name of this "all at once backward elimination" and whether it has been described/studied? $\endgroup$ – wvguy8258 Oct 25 '13 at 17:51
  • 1
    $\begingroup$ I recalled it getting a brief mention here: Colton, "Avoiding mean-square error bias in Designed experiments". On p.3 "Another inappropriate model-reduction technique can result in a positive bias in the LOF-error component" et seq. $\endgroup$ – Scortchi - Reinstate Monica Oct 25 '13 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.