Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm brand new to this R thing but am unsure which model to select.

  1. I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which is the "best".

    Model 1: Var1 (p=0.03) AIC=14.978
    Model 2: Var1 (p=0.09) + Var2 (p=0.199) AIC = 12.543
    Model 3: Var1 (p=0.04) + Var2 (p=0.04) + Var3 (p=0.06) AIC= -17.09
    

    I'm inclined to go with Model #3 because it has the lowest AIC (I heard negative is ok) and the p-values are still rather low.

    I've ran 8 variables as predictors of Hatchling Mass and found that these three variables are the best predictors.

  2. My next forward stepwise I choose Model 2 because even though the AIC was slightly larger the p values were all smaller. Do you agree this is the best?

    Model 1: Var1 (p=0.321) + Var2 (p=0.162) + Var3 (p=0.163) + Var4 (p=0.222)  AIC = 25.63
    Model 2: Var1 (p=0.131) + Var2 (p=0.009) + Var3 (p=0.0056)                  AIC = 26.518
    Model 3: Var1 (p=0.258) + Var2 (p=0.0254)                                   AIC = 36.905
    

thanks!

share|improve this question
    
Could you tell us the difference between (1) and (2)? Clearly something changed, because Model 3 in (1) and Model 2 in (2) are nominally identical but the p-values and AIC differ. –  whuber Apr 4 '11 at 18:52
    
This question has been reposted two times, which means that not only we have to close them, but also the associated response(s) that were already provided to you. Could you please register your account (see the FAQ), and pay attention to StackExchange posting policy in the future? Thanks. –  chl Apr 4 '11 at 20:56
    
@whuber, I'm afraid I don't understand your question fully. It's probably my lack of statistical understanding. But to try to clarify. Model 1 has 4 variables, Model 2 has 3 variables and Model 3 has 2 variables. The variables are in the same order in every model (meaning variable one = temp in each model). I think @GaBorgulya and @djma answered my question perfectly. Variable 4 IS correlated with variable 3. AH-HA! Makes sense. thanks oodles! –  MEL Apr 7 '11 at 11:23
    
I've converted your response to the above comment. If you feel one of the current responses helped you or answered your question, don't forget to accept it, as kindly reminded by @richiemorrisroe. BTW, good to see you registered your account. –  chl Apr 8 '11 at 1:01

3 Answers 3

up vote 8 down vote accepted

AIC is a goodness of fit measure that favours smaller residual error in the model, but penalises for including further predictors and helps avoiding overfitting. In your second set of models model 1 (the one with the lowest AIC) may perform best when used for prediction outside your dataset. A possible explanation why adding Var4 to model 2 results in a lower AIC, but higher p values is that Var4 is somewhat correlated with Var1, 2 and 3. The interpretation of model 2 is thus easier.

share|improve this answer

Looking at individual p-values can be misleading. If you have variables that are collinear (have high correlation), you will get big p-values. This does not mean the variables are useless.

As a quick rule of thumb, selecting your model with the AIC criteria is better than looking at p-values.

One reason one might not select the model with the lowest AIC is when your variable to datapoint ratio is large.

Note that model selection and prediction accuracy are somewhat distinct problems. If your goal is to get accurate predictions, I'd suggest cross-validating your model by separating your data in a training and testing set.

A paper on variable selection: Stochastic Stepwise Ensembles for Variable Selection

share|improve this answer
1  
If your goal is prediction accuracy, you want to use AIC (as it minimises the expected KL divergence between the fitted model and the truth). If you want a consistent model selection procedure (fixed p, growing n), you may use, say, BIC instead. Using p-values in stepwise regression to select hypotheses is definitelly not recommended. –  emakalic Apr 4 '11 at 23:54
2  
@emakalic - just a quick note, that AIC & BIC are basically just ways of choosing which p-value to use, rather than doing something "fundamentally" different. For AIC, we have a p-value of $0.154$ and for BIC we have a p-value equivalent to $|t|>\sqrt{\log(N)}$. –  probabilityislogic Nov 4 '11 at 9:51

AIC is motivated by the estimation of the generalization error (like Mallow's CP, BIC,...). If you want the model for predictions, better use one of these criteria. If you want your model for explaining a phenomenon, use p-values.

Also, see here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.