1
$\begingroup$

I have the following 3 models:

fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)

Edit - The variable survived is a binary variable - 1 indicating a user survived beyond 10 sessions (threshold) and 0 otherwise. ascore is a value indicating activity of user and gini is the gini-simpson index. and I am intending to check if addition of "gini" produces a better fit (classifier) than just having ascore in the model.

The AICs for the models are 22280, 22132 and 21959 which may seem to indicate fit3 > fit2 > fit1

The AUC for ROC curves are 0.7447, 0.7241 and 0.7326 which may seem to indicate fit1 > fi3 > fit2

While ascore is significant in fit1 and fit2, it is not significant in fit3.

Here are my outputs:

> fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
> summary(fit1)

Call:
glm(formula = survived ~ ascore, family = binomial, data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-4.6363  -0.3987  -0.3587  -0.3521   2.3751  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -2.7638342  0.0227365 -121.56   <2e-16 ***
ascore       0.0047660  0.0001223   38.98   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22276  on 39851  degrees of freedom
AIC: 22280

Number of Fisher Scoring iterations: 5

> fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
> summary(fit2)

Call:
glm(formula = survived ~ ascore + gini, family = binomial, data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-4.6139  -0.4180  -0.3821  -0.3302   2.6274  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.4243084  0.0638588  -53.62   <2e-16 ***
ascore       0.0048883  0.0001236   39.55   <2e-16 ***
gini         1.1661666  0.1006312   11.59   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22126  on 39850  degrees of freedom
AIC: 22132

Number of Fisher Scoring iterations: 5

> fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)
> summary(fit3)

Call:
glm(formula = survived ~ ascore + gini + ascore:gini, family = binomial, 
data = records)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3362  -0.4048  -0.3639  -0.3277   2.5484  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.2074537  0.0621832 -51.581  < 2e-16 ***
ascore       0.0000208  0.0003766   0.055    0.956    
gini         0.6632272  0.1031948   6.427  1.3e-10 ***
ascore:gini  0.0101276  0.0007541  13.430  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 21951  on 39849  degrees of freedom
AIC: 21959

Number of Fisher Scoring iterations: 5
$\endgroup$
  • 1
    $\begingroup$ OK. But a logistic regression in and of itself does not classify anything. You also need to select a threshold, which you do not seem to have described here. $\endgroup$ – whuber Jan 6 '15 at 21:14
  • 3
    $\begingroup$ Something else. Logistic regression will estimate the log odds (or equivalently, the probability $p$) that the response will be $1$, whatever $1$ happens to mean. That probability is not a classification. In order to use it for classification, you need to pick a threshold $t$ and make the classification based on whether $p \ge t$ or $p\lt t$. You have not indicated what your value of $t$ is. How your models perform for classification depends on how you pick $t$. The performance should be assessed in terms of your loss function (giving the cost of misclassification). $\endgroup$ – whuber Jan 6 '15 at 22:10
  • 3
    $\begingroup$ But avoid thresholds if at all possible. The more gold standard measure is deviance/AIC. $\endgroup$ – Frank Harrell Jan 6 '15 at 23:07
  • 2
    $\begingroup$ Saying 0.7447 > 0.7326 > 0.7241 is useless until you have computed confidence intervals for these AUCs. $\endgroup$ – Calimo Jan 7 '15 at 8:41
  • 2
    $\begingroup$ The $c$-index (ROC area) is not sensitive enough to be used to compare two models. I use it for describing predictive discrimination of a single model. $\endgroup$ – Frank Harrell Jan 7 '15 at 13:51
1
$\begingroup$

You can test baseline model versus more large one via likelihood ratio test.

Here is a link:

http://www.ats.ucla.edu/stat/mult_pkg/faq/general/nested_tests.htm

Idea is to check if increasing variables increases model fit and if this increase is not just about random variation.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your answer. In relation to the link above, does anova do the same thing? $\endgroup$ – rk567 Jan 7 '15 at 14:58
  • 2
    $\begingroup$ This question points out the need for significant amount of time doing background reading on maximum likelihood estimation. $\endgroup$ – Frank Harrell Jan 7 '15 at 16:47
  • $\begingroup$ @rk567 It seems that it would be useful for you to study this subject as Fran Harrell wrote. $\endgroup$ – Analyst Jan 8 '15 at 6:42
  • $\begingroup$ @FrankHarrell It seems so (+1) $\endgroup$ – Analyst Jan 8 '15 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.