What is the critical p-value used by the step() function in R for stepwise regression? I assume it is 0.15, but is my assumption correct? How can I change the critical p-value?
$\begingroup$
$\endgroup$
3
asked May 11, 2014 at 12:23
2 Answers
$\begingroup$
$\endgroup$
6
As I explained in my comment on your other question, step uses AIC rather than p-values.
However, for a single variable at a time, AIC does correspond to using a p-value of 0.15 (or to be more precise, 0.1573):
Consider comparing two models, which differ by a single variable. Call the models $\cal{M}_0$ (smaller model) and $\cal{M}_1$ (larger model), and let their AIC's be $\text{AIC}_0$ and $\text{AIC}_1$ respectively.
Using the AIC criterion, you'd use the larger model if $\text{AIC}_1<\text{AIC}_0$. This will be the case if $-2\log\cal{L_0}-(-2\log\cal{L_1})>2$.
But this is simply the statistic in a likelihood ratio test. From Wilks' theorem, we'll reject the null if the statistic exceeds the upper $\alpha$ quantile of a $\chi^2_1$. So if we use a hypothesis test to choose between the smaller model and the larger, we choose the larger model when $-2\log\cal{L_0}-(-2\log\cal{L_1})>C_\alpha$.
Now $2$ lies at the 84.27 percentile of a $\chi^2_1$. Hence, if we choose the larger model when it has smaller AIC, this corresponds to rejecting the null hypothesis for a test of the additional term with a p-value of $1-0.843=0.157$, or $15.7\%$
So how do you modify it?
Easy. Change the k parameter in step from 2 to something else. You want 10% instead? Make it 2.7:
qchisq(0.10,1,lower.tail=FALSE)
[1] 2.705543
You want 2.5%? Set k=5:
qchisq(0.025,1,lower.tail=FALSE)
[1] 5.023886
and so on.
However, even though that solves your question, I advise you to pay close attention to Frank Harrell's answer on your other question, and to search out responses from a great many statisticians on other questions relating to stepwise regression here, which advice tends to be very consistently to avoid stepwise procedures in general.
-
$\begingroup$ Nice explanation. Do you know whether this would hold approximately true for the p-values of ordinary regression t-tests? $\endgroup$ Jul 29, 2014 at 16:21
-
$\begingroup$ Sorry Ben, I'm not 100% sure what "this" refers to. Are you saying "could I use the above approach where you showed how to do tests at 10% and 2.5% to do 5% tests with?" If so the answer is "obviously, yes"... but the final sentence -- which points out that even though you look like you're doing tests at some nominal level, the actual type I error rates are nothing like their nominal values. In other words, you can compute a value to use, and it should be equivalent to doing stepwise set to $\alpha=0.05$ ... but the actual significance level still won't be 5%. That's only one of ... (ctd) $\endgroup$– Glen_bJul 29, 2014 at 21:31
-
$\begingroup$ (ctd) ... a slew of serious problems with stepwise. Others include biased estimates and standard errors that are far too small. $\endgroup$– Glen_bJul 29, 2014 at 21:31
-
1$\begingroup$ Momentarily putting aside problems with stepwise model selection, I'm interested in generalizing the smaller AIC => .1573 p-value rule. The Likelihood Ratio p-value you describe is fine, but in routines like R's lm, estimate/std.err is being compared to a t-distribution. This is a different test, and I was wondering if your .1573 result might hold approximately. $\endgroup$ Jul 29, 2014 at 22:39
-
2$\begingroup$ Ben: yes, the .1573 is asymptotic (based on normal; it will only be approximately correct). I think you could compute the corresponding p-value to stepAIC for a $t$, since it will depend only on the degrees of freedom in your problem (e.g. I think it's 0.1579 for 200 df); as a result you should be able to back out the desired $k$. @Nick That's most interesting. On a first glance I don't think there's a direct connection between the calculations -- they're computing the same quantity, but for different reasons. $\endgroup$– Glen_bJul 29, 2014 at 23:12
$\begingroup$
$\endgroup$
As said above, the step function in R is based on AIC criteria. But I guess by p-value you mean alpha to enter and alpha to leave. What you can do is to use the function stepwise written by Paul Rubin and available here. As you can see you have the arguments of alpha.to.enter and alpha.to.leave that you can change. Note that this function uses the F-test or equivalently t-test to select the models. Moreover, it can handle not only stepwise regression but also forward selection and backward elimination as well if you properly define the arguments.
Descriptionpart of the help page to?stepsays, in its entirety: Select a formula-based model by AIC. $\endgroup$