4
$\begingroup$

I'm trying to model the data (not make predictions) and am NOT using lasso for this, just want to know if my plan is somewhat reasonable here:

I'm modelling for a "yes/no" response variable, so I used logistic regression and stepwiseAIC for variable selection. The results gives me 13 parameters: 8 covariates with 5 interaction terms (several parameters are not significant on their own but have a significant interaction).

When I instead used stepwise based on BIC criteria, I only got two covariates and their interaction. Much simpler of course, but the deviance increased quite a bit. Since all the parameters in the small model were also in the large one, I considered the small one to be nested, so I did the deviance test (likelihood ratio test) and it gave me a p-value of nearly 0, indicating that the larger model is better.

Am I doing it right?

$\endgroup$
4
$\begingroup$

Using variable selection procedures like minimizing AIC or BIC impacts p-values of subsequent hypothesis tests (as well as impacting other things such as bias of parameter estimates and standard errors). Testing a hypothesis based on two different model selection procedures doesn't have the nominal properties; finding statistical significance for a difference of the two models is neither surprising nor necessarily informative about the relative worth of the two models.

BIC and AIC are based on different assumptions about the situation; the two aren't consistent with each other -- if $n>7$ then using BIC to select a model will always penalize larger models more harshly than AIC.

[If you use AIC to do 1 variable-at-a-time stepwise selection, it's equivalent to doing ordinary stepwise model selection based on a significance level of 15.7%; BIC would correspond to reducing the significance level with larger sample size.]

If you're going to then use hypothesis testing to choose between them, you're essentially falling back on p-values for variable selection. AIC works out "better" one way; BIC a different way, and the likelihood ratio test a different way a again.

I don't think there's a good argument for the procedure you have adopted.

$\endgroup$
  • $\begingroup$ Okay - so no deviance test for comparing the AIC model to BIC model. Follow up question - I can use it to compare two BIC models, correct? n=800 in this case, and direction=both gave me a 3-term model compared to 8 for direction = back, so I'm wondering if I can use it for such a case... $\endgroup$ – CineyEveryday Jul 13 '15 at 8:44
  • $\begingroup$ i. correct -- you select a criterion and use that criterion. ii. No, you don't compare nested models selected via BIC or AIC using a significance test. You can just compare BIC (or if you prefer, compare AIC). $\endgroup$ – Glen_b Jul 13 '15 at 9:58
0
$\begingroup$

Short answer: yes. The likelihood ratio test is used to test the difference in goodness-of-fit between two models, and what you described would be an appropriate use of this test. This page provides some additional detail on using likelihood ratio tests on nested models.

Using BIC to perform model selection is often too restrictive when your dataset is large. This might be what's happening for you; I don't know how many observations you have. The reason is because of the way AIC vs. BIC are calculated:

AIC = 2d - 2ln⁡(L) , where d is the number of parameters.

BIC = ln(n)d - 2ln(L) , where n is the number of observations and d is the number of parameters.

As we can see, the only difference in the BIC formula is that the coefficient in front of d changed from 2 to ln(n), meaning that BIC places a much higher penalty on the model if your dataset is large, thus favoring simpler models.

You also said that you're not interested in prediction, just modeling the data. So I won't go too much into detail regarding performing cross-validation on training vs. testing sets. But I do stress that this is very important; a more complex model will always fit the training data better, but the complexity might lead to overfitting when you try to apply the model elsewhere.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.