I am trying to perform stepwise regression for variable selection in R.
In matlab, the stepwisefit function is able to work in n < p problems. Trying to use step() for such problems i get the error message AIC is -infinity for this model, so 'step' cannot proceed. Is there a way to modify parameters so i can use step function?
$\begingroup$
$\endgroup$
$\begingroup$
$\endgroup$
Use Forward feature selection
min.model <- lm(y ~ 1, data=dat)
fwd.model <- step(min.model, direction = "forward", scope = (~ x1 + x2 + ... xn))
This way the model will only add predictors until it can.
-
2$\begingroup$ The probability that this will find the right model is zero. $\endgroup$ – Frank Harrell Mar 13 '14 at 15:57
-
$\begingroup$ well, you can add a limit to the steps it will search, with
stepsso that you don't have overfitting $\endgroup$ – SamuelNLP Mar 13 '14 at 15:58 -
5$\begingroup$ The first step in stepwise selection is equivalent to univariable screening, which the literature declares to be a disaster. You are selecting a predictor because it has a low $P$-value (exactly equivalent to low AIC). A low $P$ will result from a large true effect $\beta$, a large estimated effect $\hat{\beta}$, or a low standard error. If $\beta$ or standard error is mis-estimated this will cause that feature to look stronger than it actually is. This is also called publication bias, and is nothing more than classic regression to the mean. $\endgroup$ – Frank Harrell Mar 14 '14 at 12:40
-
2$\begingroup$ Stepwise regression without penalization is almost always invalid. Look first at elastic net, then ridge regression, then lasso. Avoid feature selection if at all possible because that results in worse predictive accuracy. Global shrinkage (ridge regression; L2 norm) can work better than that. $\endgroup$ – Frank Harrell Mar 14 '14 at 16:09
-
3$\begingroup$ Feature selection fails especially in that context. You must use penalization. The methods I mentioned allow $p > n$. Note that any time $p > n$ there is no known method that has a high probability of having the list of features deemed important to replicate in new samples. $\endgroup$ – Frank Harrell Mar 14 '14 at 17:21
