# So many significant explanatory variables and so small auc

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will have a great performance, but when I make predictions for observations from the test set it appears that the auc is very small, which means that the prediction is very poor.

Can anyone suggests any explanation of such situation and maybe give some advices on how to make an impact on model performance in terms of making auc higher?

My model summary

Call:
glm(formula = cliks012 ~ hours + primaryhardwaretype + browsername +
osname + age_group_troll + plec, family = "binomial", data = train)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-0.5917  -0.1958  -0.1900  -0.1851   3.4862

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)                    -6.170726   0.127305 -48.472  < 2e-16 ***
hoursEvening                   -0.016045   0.011402  -1.407 0.159380
hoursMorning                    0.063541   0.011190   5.678 1.36e-08 ***
hoursNight                      0.122892   0.022960   5.352 8.68e-08 ***
primaryhardwaretypeMobilePhone  2.050345   0.089335  22.951  < 2e-16 ***
primaryhardwaretypeOther        0.213355   0.215761   0.989 0.322738
primaryhardwaretypeTablet       2.066480   0.124671  16.575  < 2e-16 ***
browsernameChrome               0.032222   0.099976   0.322 0.747226
browsernameFirefox              0.111669   0.099724   1.120 0.262805
browsernameInternetExplorer     0.057690   0.100636   0.573 0.566473
browsernameOther                0.187177   0.101820   1.838 0.066017 .
browsernameSafari               0.042307   0.111451   0.380 0.704241
osnameiOS                      -0.003356   0.109021  -0.031 0.975444
osnameLinux                     2.132874   0.144926  14.717  < 2e-16 ***
osnameLinuxUbuntu               2.354108   0.147236  15.989  < 2e-16 ***
osnameOSX                       2.351537   0.138428  16.987  < 2e-16 ***
osnameOther                     2.110826   0.107162  19.698  < 2e-16 ***
osnameWindows7                  2.168022   0.132195  16.400  < 2e-16 ***
osnameWindows8                  2.103507   0.135373  15.539  < 2e-16 ***
osnameWindows81                 2.197786   0.132560  16.579  < 2e-16 ***
osnameWindowsVista              2.136239   0.133530  15.998  < 2e-16 ***
osnameWindowsXP                 2.121562   0.132499  16.012  < 2e-16 ***
age_group_trollOldTroll         0.008260   0.052773   0.157 0.875619
age_group_trollWorker          -0.097919   0.015123  -6.475 9.49e-11 ***
age_group_trollYoung           -0.109535   0.021494  -5.096 3.47e-07 ***
age_group_trollYoungTroll      -0.067748   0.087009  -0.779 0.436197
plecM                           0.045470   0.012042   3.776 0.000159 ***
plecO                           0.071644   0.024581   2.915 0.003561 **
plecX                          -0.109337   0.052179  -2.095 0.036135 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 490133  on 2699999  degrees of freedom
Residual deviance: 488984  on 2699971  degrees of freedom
AIC: 489042

Number of Fisher Scoring iterations: 7


And and auc computations

library(ROCR)
auc <- function(pred_probs, real_classes){
pred <- prediction(pred_probs, real_classes)
performance(pred, "auc")@y.values[[1]]
}

preds <- predict(modelGLM_clicks012, newdata = test, type="response")
> auc(preds, test\$cliks012)
[1] 0.5230328

• How many observations do you have? – lanenok Apr 30 '15 at 15:52
• 2,7 mln in Train set and 2,76 mln in test set – Marcin Kosiński Apr 30 '15 at 16:01

Although there are many "significant" explanatory variables in your model, note in the summary of your glm model that the deviance was barely diminished by that model. (Residual deviance was almost the same as the null-model deviance.) With millions of cases as you have here, explanatory variables can be "significant" in terms of the standard criterion of p < 0.05 but still not be very important in terms of explaining the distribution of the observed dependent variable. The low AUC, at first glance, seems completely consistent with the glm output.