5
$\begingroup$

I am modeling claim severity by GLM in r. I want to check whether all predictors are significant. I use the likelihood ratio test (by the function anova()) in turn for all predictors. Finally, I get the reduced model

> anova(severity,model.severity,test="LRT")

Model 1: claimcst0 ~ gender + area + agecat
Model 2: claimcst0 ~ veh_value + veh_age + veh_body + gender + area + 
agecat
  Resid. Df Resid. Dev Df Deviance Pr(>Chi)  
1      4612     7558.0                       
2      4596     7488.7 16   69.371  0.09736 .

We can see the p-value of the test is greater than 0.05. We do not reject the smaller model.

But the funny thing is that the smaller model has greater residual deviance and AIC.

 model A
 Residual deviance: 7558.0  on 4612  degrees of freedom
 AIC: 85079 

 model B
 Residual deviance: 7488.7  on 4596  degrees of freedom
 AIC: 85055

What is the reason for it? Is the model A still the better model compared to model B?

$\endgroup$
1
  • 1
    $\begingroup$ What do you mean by 'reduced model'? If you excluded 'insignificant' predictors you may invalidate everything about the model. $\endgroup$ – Frank Harrell May 14 '18 at 14:09
4
$\begingroup$

You do not reject model 2 because there is no effect, but because the effect is not significant enough.

The results are consistent.

Note that an insignificant result for an anova model can still acknowledge an insignificant difference between the two models, e.g. that the likelihood ratio for the small and big model is in favor of the big model with the lower aic or higher relative likelihood (even when the statistiscian/experimenter chooses to prefer the small model).

It is just that anova uses a conservative (different) decision criteria (the arbitrary low p<0.05) that is

  • not about deciding model 1 over model 2 or the other way around
  • but instead about deciding whether there is a significant difference between model 1 and model 2 (and in case of no significant difference the simpler model 1 is kept).

The choice to prefer model 1 over model 2 when the anova outcome indicates that there is no large significant difference is entirely due to the statistician's (subjective) use of the anova test and not due to the anova test itselve.

The anova model does not indicate that you do not have model 2 > model 1 (or model B > model A), it only indicates that the observed difference (model2 > model 1) is not significant (where the meaning of significance is very subjective and arbitrarily set at some p-value cut-off level). The observed 'advantage'/'difference' for model B may happen by chance. That is: in the case that there is no difference (null hypothesis) then you would get occasionally (9.736% of the time) such a similar higher aic, residual deviance, likelihood ratio or whatever.

$\endgroup$
0
3
$\begingroup$

The residual deviance will always be smaller (or at least equal to) for the larger model. Similar to adding variables will only increase R-Squared in regression.

AIC is only slightly better for Model B, I bet BIC is actually worse as it more severely punishes additional variables.

Different model fit statistics will not always agree. Here it looks like 2 out of the 3 will select the smaller model (assuming BIC is smaller for Model A). The end choice on what model to select is up to you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.