5
$\begingroup$

I'm curious about how to understand the accuracy of my model which I computed with glm( family = binomial(logit) ).

In some articles it is mentioned that we should perform chisq test with residual deviance with it's DoF. When I call summary() of my glm module. "Residual deviance: 9109.9 on 99993 degrees of freedom" Therefore when I perform pchisq test with these inputs: 1-pchisq(9110, 99993) it returns 1.

Hence it is much more greater than our significance level. So we are curious about why does it return 1, is it a perfect model ?

In addition to these, here's the output of my Logistic Regression Model

Logistic Regression Model

lrm(formula = bool.revenue.all.time ~ level + building.count + 
    gold.spent + npc + friends + post.count, data = sn, x = TRUE, 
    y = TRUE)

                      Model Likelihood     Discrimination    Rank Discrim.    
                         Ratio Test            Indexes          Indexes       
Obs         1e+05    LR chi2    1488.63    R2       0.147    C       0.774    
 0          99065    d.f.             6    g        1.141    Dxy     0.547    
 1            935    Pr(> chi2) <0.0001    gr       3.130    gamma   0.586    
max |deriv| 8e-09                          gp       0.011    tau-a   0.010    
                                           Brier    0.009                     

               Coef    S.E.   Wald Z Pr(>|Z|)
Intercept      -6.7910 0.0938 -72.36 <0.0001 
level           0.0756 0.0193   3.92 <0.0001 
building.count  0.0698 0.0091   7.64 <0.0001 
gold.spent      0.0020 0.0002  11.05 <0.0001 
npc             0.0172 0.0057   3.03 0.0024  
friends         0.0304 0.0045   6.82 <0.0001 
post.count     -0.0132 0.0042  -3.17 0.0015

This is validation with bootstrap's output

  index.orig training   test optimism index.corrected    n
Dxy           0.5511   0.5500 0.5506  -0.0006          0.5518 1000
R2            0.1469   0.1469 0.1465   0.0005          0.1465 1000
Intercept     0.0000   0.0000 0.0002  -0.0002          0.0002 1000
Slope         1.0000   1.0000 0.9997   0.0003          0.9997 1000
Emax          0.0000   0.0000 0.0001   0.0001          0.0001 1000
D             0.0149   0.0149 0.0148   0.0000          0.0148 1000
U             0.0000   0.0000 0.0000   0.0000          0.0000 1000
Q             0.0149   0.0149 0.0148   0.0001          0.0148 1000
B             0.0086   0.0086 0.0086   0.0000          0.0086 1000
g             1.1410   1.1381 1.1365   0.0016          1.1394 1000
gp            0.0111   0.0111 0.0111   0.0000          0.0111 1000

And this is the output of my calibration curve:

n=100000   Mean absolute error=0.002   Mean squared error=5e-05
0.9 Quantile of absolute error=0.002

Calibration Curve

Thanks.

Improve this question
$\endgroup$
11
  • 2
    $\begingroup$ What do you mean by "accuracy"? Correctness of prediction? Sensitivity? Specificity? Something else? $\endgroup$
    – Peter Flom
    Jul 23, 2013 at 10:54
  • 1
    $\begingroup$ Hi, I mean, Correctness of prediction. To understand how reliable it is. $\endgroup$
    – CanCeylan
    Jul 23, 2013 at 11:00
  • 2
    $\begingroup$ That test does not have a $\chi^2$ distribution but rather a degenerate distribution because the degrees of freedom increases with $n$. To better understand accuracy, look at measures of calibration and predictive accuracy. See stats.stackexchange.com/questions/64788/… for a way to obtain multiple interesting estimates. $\endgroup$
    – Frank Harrell
    Jul 23, 2013 at 11:23
  • 3
    $\begingroup$ The $c$-index does not address calibration at all. The calibration curve certainly looks odd. What are the 100th lowest and highest predictions? Sometimes we need to not focus on the part of the calibration curve not supported by a sufficient sample size. On the other hand you could have a misspecified model. $\endgroup$
    – Frank Harrell
    Jul 23, 2013 at 15:15
  • 2
    $\begingroup$ f <- lrm(); n <- sum(f$freq); quantile(predict(f, type='fitted'), c(100/n, 1-100/n), na.rm=TRUE) $\endgroup$
    – Frank Harrell
    Jul 23, 2013 at 15:23

2 Answers 2

Reset to default
6
$\begingroup$

If you want to assess accuracy, one way is to look at the predicted outcome vs. the actual outcome. You can get the predicted values with fitted-values and then compare them to the actual values; for one example see this page:

Improve this answer
$\endgroup$
1
  • $\begingroup$ What is the difference between comparing the fitted vs. actual values of a logistic regression and calculating the predicted probabilities on a training data set and using them to test the predictive accuracy on a testing data set? $\endgroup$
    – coip
    Feb 16, 2018 at 0:00
-2
$\begingroup$

Try the below code:

currMod <- glm(as.formula(form),family=binomial(link='logit'), 
  data=inputData_signif_training)  # build the model

predictedval <- predict(currMod,newdata=inputData_signif_test[1:6],type='response')

fitted.results.cat <- ifelse(predictedval > 0.5,"Yes","No")

fitted.results.cat<-as.factor(fitted.results.cat)

require(caret)    
cm<-confusionMatrix(data=fitted.results.cat, 
  reference=inputData_signif_test$Heart.Failure)

Accuracy<-round(cm$overall[1],2)
Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Those are improper accuracy scoring rules requiring highly arbitrary binning of continuous probabilities. This problem was already solved using high-resolution calibration curves. $\endgroup$
    – Frank Harrell
    Mar 22, 2016 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.