2
$\begingroup$

Have you ever seen a model with almost every significant variable and such small auc (area under the ROC curve) ? What might be the cause of it? When I saw summary of a model I thought this model will have a great performance, but when I make predictions for observations from the test set it appears that the auc is very small, which means that the prediction is very poor.

Can anyone suggests any explanation of such situation and maybe give some advices on how to make an impact on model performance in terms of making auc higher?

My model summary

Call:
glm(formula = cliks012 ~ hours + primaryhardwaretype + browsername + 
    osname + age_group_troll + plec, family = "binomial", data = train)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.5917  -0.1958  -0.1900  -0.1851   3.4862  

Coefficients:
                                Estimate Std. Error z value Pr(>|z|)    
(Intercept)                    -6.170726   0.127305 -48.472  < 2e-16 ***
hoursEvening                   -0.016045   0.011402  -1.407 0.159380    
hoursMorning                    0.063541   0.011190   5.678 1.36e-08 ***
hoursNight                      0.122892   0.022960   5.352 8.68e-08 ***
primaryhardwaretypeMobilePhone  2.050345   0.089335  22.951  < 2e-16 ***
primaryhardwaretypeOther        0.213355   0.215761   0.989 0.322738    
primaryhardwaretypeTablet       2.066480   0.124671  16.575  < 2e-16 ***
browsernameChrome               0.032222   0.099976   0.322 0.747226    
browsernameFirefox              0.111669   0.099724   1.120 0.262805    
browsernameInternetExplorer     0.057690   0.100636   0.573 0.566473    
browsernameOther                0.187177   0.101820   1.838 0.066017 .  
browsernameSafari               0.042307   0.111451   0.380 0.704241    
osnameiOS                      -0.003356   0.109021  -0.031 0.975444    
osnameLinux                     2.132874   0.144926  14.717  < 2e-16 ***
osnameLinuxUbuntu               2.354108   0.147236  15.989  < 2e-16 ***
osnameOSX                       2.351537   0.138428  16.987  < 2e-16 ***
osnameOther                     2.110826   0.107162  19.698  < 2e-16 ***
osnameWindows7                  2.168022   0.132195  16.400  < 2e-16 ***
osnameWindows8                  2.103507   0.135373  15.539  < 2e-16 ***
osnameWindows81                 2.197786   0.132560  16.579  < 2e-16 ***
osnameWindowsVista              2.136239   0.133530  15.998  < 2e-16 ***
osnameWindowsXP                 2.121562   0.132499  16.012  < 2e-16 ***
age_group_trollOldTroll         0.008260   0.052773   0.157 0.875619    
age_group_trollWorker          -0.097919   0.015123  -6.475 9.49e-11 ***
age_group_trollYoung           -0.109535   0.021494  -5.096 3.47e-07 ***
age_group_trollYoungTroll      -0.067748   0.087009  -0.779 0.436197    
plecM                           0.045470   0.012042   3.776 0.000159 ***
plecO                           0.071644   0.024581   2.915 0.003561 ** 
plecX                          -0.109337   0.052179  -2.095 0.036135 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 490133  on 2699999  degrees of freedom
Residual deviance: 488984  on 2699971  degrees of freedom
AIC: 489042

Number of Fisher Scoring iterations: 7

And and auc computations

library(ROCR)
auc <- function(pred_probs, real_classes){
  pred <- prediction(pred_probs, real_classes)
  performance(pred, "auc")@y.values[[1]] 
}

preds <- predict(modelGLM_clicks012, newdata = test, type="response")
> auc(preds, test$cliks012)
[1] 0.5230328
$\endgroup$
  • $\begingroup$ How many observations do you have? $\endgroup$ – lanenok Apr 30 '15 at 15:52
  • $\begingroup$ 2,7 mln in Train set and 2,76 mln in test set $\endgroup$ – Marcin Kosiński Apr 30 '15 at 16:01
4
$\begingroup$

Although there are many "significant" explanatory variables in your model, note in the summary of your glm model that the deviance was barely diminished by that model. (Residual deviance was almost the same as the null-model deviance.) With millions of cases as you have here, explanatory variables can be "significant" in terms of the standard criterion of p < 0.05 but still not be very important in terms of explaining the distribution of the observed dependent variable. The low AUC, at first glance, seems completely consistent with the glm output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.