# Use likelihood-ratio test to select models in case of nested models

Is it possible to do model selection in this way? Suppose I need to select a good (logistic) model among three variables (var1, var2, var3). The deviance D* (-2*log-likelihood) of this full model would be the minimum among all possible models. Then I could try all 6 combination of sub-models(1,2,3,12,13,23) and compute their deviance D1~D6. Next I compute the difference: deltaD_i=D_i-D*, this should follow chi-square distribution with df=differences_in_variables_numbers. The models with deltaD within 95% confidence interval of D* would be within the confidence interval of the full model, that is, the variance explained by the reduced model is not significantly different than the full model. Then we could accept these model as good models. By doing this, we could end of with several "good" models.

Is this somehow a possible way to do model selection?

Thanks

-
The short answer is that "yes, this is a possible way". It may wind up, as you say, with several models. That can be a good thing. In many areas, there is no one "true" model. There are techniques like model averaging that can combine models (at some cost in interpretability, in my opinion). A good book on this is Model Selection and Multimodel Averaging by Burnham and Anderson. –  Peter Flom Aug 31 '12 at 14:50
@PeterFlom That is a good answer and a good book recommendation. –  Michael Chernick Aug 31 '12 at 15:08

Normally looking at all subsets as you do in your example is not done because there are so many. In your example with just three variables it can be done and would usually be done. Rather than use the deviance which gives the smallest value to the full model, criteria like AIC or BIC are used as they penalize the loglikelihood for using many parameters. Models are chosen that minimize the criterion (e.g. AIC, BIC). Looking at variance explained is using R square as the criterion. Minimizing adjusted R square is sometimes used in regression for the same reason as the peanlized likelihood measures (i.e. R square is maximized when using the full model).

Your idea that you decide a model is good if the extra variance explained by the full model is not statistically significantly higher than with the given model is a sensible way to call a subset of models "good".

-

Here's a somewhat longer answer, to complement my comment and @MichaelChernick's answer.

You can do this. With only 6 models - sure.

BUT

1. What about interactions? Even with only 2 way interactions, your number of possible models goes up fast. You now have null model, 1, 2, 3, 1 + 2, 1 + 3, 2 + 3, 1 + 2 + 3 (which you had before) plus 1 2 12, 1 3 13, 2 3 23, 1 2 3 12, 1 2 3 13, 1 2 3 23. Are these important to look at? It gets hairy, fast, to look at all possible subsets (esp. because you should impose hierarchy).

2. Some variables may need to be in the model, just because of the nature of the field. If a certain effect is well-established, then finding a small (and perhaps non-significant) effect may be more important than finding a large one. As an example, suppose you found that, in some isolated tribe of humans, men and women were not much different on height. You would want to know! You certainly wouldn't want to throw that variable out because of non-significance or small size.

3. Some variables may be important because of their effect on other variables' coefficients. (I am actually not sure how this plays out with log-likelihood, AIC, BIC etc).

4. Finally, using ANY automated scheme (even the best) does one very bad thing. It allows you not to think. Of course, you can use automated method to generate thought - that's good. But using them as black boxes is not something I can recommend.

-
+10 on item 4 :) –  StasK Aug 31 '12 at 16:43