# how to decide which logistic regression model is better?

I have the following 3 models:

fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)


Edit - The variable survived is a binary variable - 1 indicating a user survived beyond 10 sessions (threshold) and 0 otherwise. ascore is a value indicating activity of user and gini is the gini-simpson index. and I am intending to check if addition of "gini" produces a better fit (classifier) than just having ascore in the model.

The AICs for the models are 22280, 22132 and 21959 which may seem to indicate fit3 > fit2 > fit1

The AUC for ROC curves are 0.7447, 0.7241 and 0.7326 which may seem to indicate fit1 > fi3 > fit2

While ascore is significant in fit1 and fit2, it is not significant in fit3.

Here are my outputs:

> fit1 <- glm(formula = survived ~ ascore, data=records, family = binomial)
> summary(fit1)

Call:
glm(formula = survived ~ ascore, family = binomial, data = records)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-4.6363  -0.3987  -0.3587  -0.3521   2.3751

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.7638342  0.0227365 -121.56   <2e-16 ***
ascore       0.0047660  0.0001223   38.98   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22276  on 39851  degrees of freedom
AIC: 22280

Number of Fisher Scoring iterations: 5

> fit2 <- glm(formula = survived ~ ascore + gini, data=records, family = binomial)
> summary(fit2)

Call:
glm(formula = survived ~ ascore + gini, family = binomial, data = records)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-4.6139  -0.4180  -0.3821  -0.3302   2.6274

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.4243084  0.0638588  -53.62   <2e-16 ***
ascore       0.0048883  0.0001236   39.55   <2e-16 ***
gini         1.1661666  0.1006312   11.59   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 22126  on 39850  degrees of freedom
AIC: 22132

Number of Fisher Scoring iterations: 5

> fit3 <- glm(formula = survived ~ ascore + gini + ascore:gini, data=records, family = binomial)
> summary(fit3)

Call:
glm(formula = survived ~ ascore + gini + ascore:gini, family = binomial,
data = records)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-2.3362  -0.4048  -0.3639  -0.3277   2.5484

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -3.2074537  0.0621832 -51.581  < 2e-16 ***
ascore       0.0000208  0.0003766   0.055    0.956
gini         0.6632272  0.1031948   6.427  1.3e-10 ***
ascore:gini  0.0101276  0.0007541  13.430  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 23752  on 39852  degrees of freedom
Residual deviance: 21951  on 39849  degrees of freedom
AIC: 21959

Number of Fisher Scoring iterations: 5

• OK. But a logistic regression in and of itself does not classify anything. You also need to select a threshold, which you do not seem to have described here. – whuber Jan 6 '15 at 21:14
• Something else. Logistic regression will estimate the log odds (or equivalently, the probability $p$) that the response will be $1$, whatever $1$ happens to mean. That probability is not a classification. In order to use it for classification, you need to pick a threshold $t$ and make the classification based on whether $p \ge t$ or $p\lt t$. You have not indicated what your value of $t$ is. How your models perform for classification depends on how you pick $t$. The performance should be assessed in terms of your loss function (giving the cost of misclassification). – whuber Jan 6 '15 at 22:10
• But avoid thresholds if at all possible. The more gold standard measure is deviance/AIC. – Frank Harrell Jan 6 '15 at 23:07
• Saying 0.7447 > 0.7326 > 0.7241 is useless until you have computed confidence intervals for these AUCs. – Calimo Jan 7 '15 at 8:41
• The $c$-index (ROC area) is not sensitive enough to be used to compare two models. I use it for describing predictive discrimination of a single model. – Frank Harrell Jan 7 '15 at 13:51