26
$\begingroup$

I'm brand new to this R thing but am unsure which model to select.

  1. I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the "best".

    Model 1: Var1 (p=0.03) AIC=14.978
Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543
Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09

    I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p-values are still rather low.

    I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.

  2. My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best?

    Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222)  AIC = 25.63
Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056)                  AIC = 26.518
Model 3: Var1 (p=0.258) + Var2 (p=0.0254)                                   AIC = 36.905

thanks!

Improve this question
$\endgroup$
4
  • $\begingroup$ Could you tell us the difference between (1) and (2)? Clearly something changed, because Model 3 in (1) and Model 2 in (2) are nominally identical but the p-values and AIC differ. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2011 at 18:52
  • 2
    $\begingroup$ This question has been reposted two times, which means that not only we have to close them, but also the associated response(s) that were already provided to you. Could you please register your account (see the FAQ), and pay attention to StackExchange posting policy in the future? Thanks. $\endgroup$
    – chl
    Commented Apr 4, 2011 at 20:56
  • $\begingroup$ @whuber, I'm afraid I don't understand your question fully. It's probably my lack of statistical understanding. But to try to clarify. Model 1 has 4 variables, Model 2 has 3 variables and Model 3 has 2 variables. The variables are in the same order in every model (meaning variable one = temp in each model). I think @GaBorgulya and @djma answered my question perfectly. Variable 4 IS correlated with variable 3. AH-HA! Makes sense. thanks oodles! $\endgroup$
    – MEL
    Commented Apr 7, 2011 at 11:23
  • $\begingroup$ I've converted your response to the above comment. If you feel one of the current responses helped you or answered your question, don't forget to accept it, as kindly reminded by @richiemorrisroe. BTW, good to see you registered your account. $\endgroup$
    – chl
    Commented Apr 8, 2011 at 1:01

3 Answers 3

Reset to default
25
$\begingroup$

AIC is a goodness of fit measure that favours smaller residual error in the model, but penalises for including further predictors and helps avoiding overfitting. In your second set of models model 1 (the one with the lowest AIC) may perform best when used for prediction outside your dataset. A possible explanation why adding Var4 to model 2 results in a lower AIC, but higher p values is that Var4 is somewhat correlated with Var1, 2 and 3. The interpretation of model 2 is thus easier.

Improve this answer
$\endgroup$
36
$\begingroup$

Looking at individual p-values can be misleading. If you have variables that are collinear (have high correlation), you will get big p-values. This does not mean the variables are useless.

As a quick rule of thumb, selecting your model with the AIC criteria is better than looking at p-values.

One reason one might not select the model with the lowest AIC is when your variable to datapoint ratio is large.

Note that model selection and prediction accuracy are somewhat distinct problems. If your goal is to get accurate predictions, I'd suggest cross-validating your model by separating your data in a training and testing set.

A paper on variable selection: Stochastic Stepwise Ensembles for Variable Selection

Improve this answer
$\endgroup$
7
  • 5
    $\begingroup$ If your goal is prediction accuracy, you want to use AIC (as it minimises the expected KL divergence between the fitted model and the truth). If you want a consistent model selection procedure (fixed p, growing n), you may use, say, BIC instead. Using p-values in stepwise regression to select hypotheses is definitelly not recommended. $\endgroup$
    – emakalic
    Commented Apr 4, 2011 at 23:54
  • 9
    $\begingroup$ @emakalic - just a quick note, that AIC & BIC are basically just ways of choosing which p-value to use, rather than doing something "fundamentally" different. For AIC, we have a p-value of $0.154$ and for BIC we have a p-value equivalent to $|t|>\sqrt{\log(N)}$. $\endgroup$
    – probabilityislogic
    Commented Nov 4, 2011 at 9:51
  • $\begingroup$ That's nice to know. I have a question wrt to using the p-value equivalent of BIC in a step-wise regression. Would this be a static value I. E, do I use the initial p-value set for N - k - 1 degrees of freedom. Or does this get updated dynamically at each step, with the addition or subtraction of the degrees of freedom? Does it even matter? $\endgroup$
    – Linus Fernandes
    Commented May 3, 2020 at 20:45
  • $\begingroup$ @probabilityislogic, can you give any reference that demonstrates what you say regarding the p-value of 0.154? I am having trouble seeing this ... $\endgroup$
    – koteletje
    Commented Sep 22, 2020 at 10:01
  • 2
    $\begingroup$ @koteletje - Bayesian model selection in social research by Adrian raftery. social methodology, vol25 (1995), 111-163 equation (20) has the connection with the lr chi square statistic $\endgroup$
    – probabilityislogic
    Commented Sep 23, 2020 at 10:43
-1
$\begingroup$

AIC is motivated by the estimation of the generalization error (like Mallow's CP, BIC,...). If you want the model for predictions, better use one of these criteria. If you want your model for explaining a phenomenon, use p-values.

Also, see here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.