9
$\begingroup$

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154.

I tried it in R, where I used a "backward" subset selection algorithm to throw out variables from a full specification. First, by sequentially throwing out the variable with the highest p-value and stopping when all p-values are below 0.154 and, secondly, by dropping the variable which results in lowest AIC when removed until no improvement can be made.

It turned out that they give roughly the same results when I use a p-value of 0.154 as threshold.

Is this actually true? If so, does anyone know why or can refer to a source which explains it?

P.S. I couldn't ask the person commenting or write a comment, because just signed up. I am aware that this is not the most suitable approach to model selection and inference etc.

$\endgroup$
  • $\begingroup$ (1) Prognostic modeling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Statistics in Medicine, 19, 1059-1079 (2) true for variables with df1, based on aic definition. But could be lower if your degrees of freedom of variables higher $\endgroup$ – charles Mar 7 '14 at 21:35
13
$\begingroup$

Variable selection done using statistical testing or AIC is highly problematic. If using $\chi^2$ tests, AIC uses a cutoff of $\chi^2$=2.0 which corresponds to $\alpha=0.157$. AIC when used on individual variables does nothing new; it just uses a more reasonable $\alpha$ than 0.05. A more reasonable (less inference-disturbing) $\alpha$ is 0.5.

$\endgroup$
  • $\begingroup$ +1 I spent so long constructing my (now deleted) answer, I didn't even see this one had posted in the meantime. I would have just voted this one up instead. $\endgroup$ – Glen_b -Reinstate Monica Mar 7 '14 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.