Let's assume $p$ potential predictor variables $X_1,...,X_p$ and a single dependent variable $Y$.

Now I evaluate the performance of all possible linear models considering all possible combinations of predictor variables ($2^p-1$). The performance measure(s) could be pretty much any statistic but first comes to mind $R^2$, $F$-statistic and MSE. Based on them I select the "best" model or the top selection which I can check out more closely.

Intuitively I would assume this is a great idea—but I read around a bit and came across the infamous concept of "stepwise regression" and how it is considered useless by a lot of ( though apparently not all) statistically trained people. The reason seems to be that the assumed distribution underlying the involved statistics does not hold for the scenario.

But stepwise regression is usually described as a slightly different algorithm where you start with a model, adding and removing variables from the model based on a criterion for a statistic.

So my question is whether the approach I describe would also be a type of stepwise regression and hence be handicapped by design. On the latter part (if it is SwReg) I would be interested in clarification on where the handicap comes into play and whether it is possible to amend it.

  • $\begingroup$ Your approach is called "full subset selection". $\endgroup$ – Michael M Nov 8 '13 at 9:17
  • 1
    $\begingroup$ Yes, it shares the data snooping properties of stepwise selection. In fact, stepwise selection is an efficient approximation to full subset selection. $\endgroup$ – Michael M Nov 8 '13 at 9:21
  • 2
    $\begingroup$ Your original notation of $N, n$ for number of predictors was a minor but still unnecessary distraction (inconsistent too), given their much common use for number of cases or observations. I took the liberty of editing it out. $\endgroup$ – Nick Cox Nov 8 '13 at 9:43
  • 1
    $\begingroup$ Out of the measures you mentioned, $R^2$ and MSE do not seem appropriate to me. The full model (the one including all $p$ regressors) will always give you the highest $R^2$ and the lowest MSE, regardless what the data generating process was. Meanwhile, $F$-statistic may or may not be appropriate, I am not sure. $\endgroup$ – Richard Hardy Nov 2 '15 at 15:38
  • $\begingroup$ I don't know about "stepwise", and I don't think it qualifies as subset selection, but the Noam Ross approach with glmulti is what I might look at for a starter. noamross.net/blog/2013/2/20/model-selection-drug.html $\endgroup$ – EngrStudent - Reinstate Monica Nov 2 '15 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.