2
$\begingroup$

I have the following 3 models:

fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)

Edit - The variable survived is a binary variable - 1 indicating a user survived beyond 10 sessions (threshold) and 0 otherwise. ascore is a value indicating activity of user and gini is the gini-simpson index. and I am intending to check if addition of "gini" produces a better fit (classifier) than just having ascore in the model.

The AICs for the models are 22280, 22132 and 21959 which may seem to indicate fit3 > fit2 > fit1

The AUC for ROC curves are 0.7447, 0.7241 and 0.7326 which may seem to indicate fit1 > fi3 > fit2

While ascore is significant in fit1 and fit2, it is not significant in fit3.

Here are my outputs:

> fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
> summary(fit1)

Call:
glm(formula = survived ~ ascore, family = binomial, data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-4.6363  -0.3987  -0.3587  -0.3521   2.3751  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -2.7638342  0.0227365 -121.56   <2e-16 ***
ascore       0.0047660  0.0001223   38.98   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22276  on 39851  degrees of freedom
AIC: 22280

Number of Fisher Scoring iterations: 5

> fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
> summary(fit2)

Call:
glm(formula = survived ~ ascore + gini, family = binomial, data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-4.6139  -0.4180  -0.3821  -0.3302   2.6274  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.4243084  0.0638588  -53.62   <2e-16 ***
ascore       0.0048883  0.0001236   39.55   <2e-16 ***
gini         1.1661666  0.1006312   11.59   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22126  on 39850  degrees of freedom
AIC: 22132

Number of Fisher Scoring iterations: 5

> fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)
> summary(fit3)

Call:
glm(formula = survived ~ ascore + gini + ascore:gini, family = binomial, 
data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3362  -0.4048  -0.3639  -0.3277   2.5484  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.2074537  0.0621832 -51.581  < 2e-16 ***
ascore       0.0000208  0.0003766   0.055    0.956    
gini         0.6632272  0.1031948   6.427  1.3e-10 ***
ascore:gini  0.0101276  0.0007541  13.430  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 21951  on 39849  degrees of freedom
AIC: 21959

Number of Fisher Scoring iterations: 5
Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ OK. But a logistic regression in and of itself does not classify anything. You also need to select a threshold, which you do not seem to have described here. $\endgroup$
    – whuber ♦
    Jan 6, 2015 at 21:14
  • 3
    $\begingroup$ Something else. Logistic regression will estimate the log odds (or equivalently, the probability $p$) that the response will be $1$, whatever $1$ happens to mean. That probability is not a classification. In order to use it for classification, you need to pick a threshold $t$ and make the classification based on whether $p \ge t$ or $p\lt t$. You have not indicated what your value of $t$ is. How your models perform for classification depends on how you pick $t$. The performance should be assessed in terms of your loss function (giving the cost of misclassification). $\endgroup$
    – whuber ♦
    Jan 6, 2015 at 22:10
  • 3
    $\begingroup$ But avoid thresholds if at all possible. The more gold standard measure is deviance/AIC. $\endgroup$
    – Frank Harrell
    Jan 6, 2015 at 23:07
  • 2
    $\begingroup$ Saying 0.7447 > 0.7326 > 0.7241 is useless until you have computed confidence intervals for these AUCs. $\endgroup$
    – Calimo
    Jan 7, 2015 at 8:41
  • 2
    $\begingroup$ The $c$-index (ROC area) is not sensitive enough to be used to compare two models. I use it for describing predictive discrimination of a single model. $\endgroup$
    – Frank Harrell
    Jan 7, 2015 at 13:51

1 Answer 1

Reset to default
1
$\begingroup$

You can test baseline model versus more large one via likelihood ratio test.

Here is a link:

http://www.ats.ucla.edu/stat/mult_pkg/faq/general/nested_tests.htm

Idea is to check if increasing variables increases model fit and if this increase is not just about random variation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. In relation to the link above, does anova do the same thing? $\endgroup$
    – rk567
    Jan 7, 2015 at 14:58
  • 2
    $\begingroup$ This question points out the need for significant amount of time doing background reading on maximum likelihood estimation. $\endgroup$
    – Frank Harrell
    Jan 7, 2015 at 16:47
  • $\begingroup$ @rk567 It seems that it would be useful for you to study this subject as Fran Harrell wrote. $\endgroup$
    – Analyst
    Jan 8, 2015 at 6:42
  • $\begingroup$ @FrankHarrell It seems so (+1) $\endgroup$
    – Analyst
    Jan 8, 2015 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.