2
$\begingroup$

I have compared two logistic regression models using the function anova(mod1,mod2,test="Chisq") in R. The result that I obtained is the following:

Model 1: Status ~ Added.genes.var
Model 2: Status ~ Added.genes.var + mult_genes
Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
1       887     1218.0                          
2       886     1184.2  1   33.805 6.093e-09 ***

As far as I understand, this means that adding the 'mult_genes' predictor variable to the model, significantly improves its fit. For this reason, when checking the area under the curve (AUC) with the function auc() of the package pROC, I was expecting to see a significant difference in that as well, but I obtained, for model 1 and 2 respectively:

Area under the curve: 0.6147
Area under the curve: 0.6158

I believe 0.61 is quite low as AUC...does this mean that both models perform poorly? And how is it possible to get such a significant result when comparing models with anova(), and so similar AUCs at the same time?
Thanks a lot!

$\endgroup$
2
$\begingroup$

"Significant" in the context of your hypothesis test means that if the coefficient for mult_genes were really zero (i.e. mult_genes had no effect on Status) the chances of estimating that coefficient to be as big or bigger than the estimate you made are small. That's not the same as saying it makes a big difference to the predictions over its range in your sample, given that you've included Added.genes.var. So there's no contradiction between its being "significant" & only making a small difference to the AUC.

As to the AUC's being poor or otherwise, that depends entirely on what you intend to do with the model. For some applications it might be fine; for others, hopeless: have you got a better model up your sleeve, do other people have better models, could you reasonably expect to get a better model?

$\endgroup$
2
  • $\begingroup$ Thank you for your answer! So, doesn't the p-value of 6.093e-9 with a reduced residual deviance for model 2 indicate that its fit is significantly improved with respect to model 1? If not, which parameters should I check to see if 'mult_genes' significantly improves my model? Please tell me if I am missing something... $\endgroup$
    – Franz
    Jan 23 '14 at 10:18
  • $\begingroup$ It does indicate that: just don't confuse "significantly improved" with "greatly improved". Someone here suggested "discernible" would have been a better choice than "significant" as a statistical term: so think "the fit of Model 2 is discernibly improved with respect to Model 1". $\endgroup$ Jan 23 '14 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.