# How to determine the accuracy of logistic regression in R?

I'm curious about how to understand the accuracy of my model which I computed with glm( family = binomial(logit) ).

In some articles it is mentioned that we should perform chisq test with residual deviance with it's DoF. When I call summary() of my glm module. "Residual deviance: 9109.9 on 99993 degrees of freedom" Therefore when I perform pchisq test with these inputs: 1-pchisq(9110, 99993) it returns 1.

Hence it is much more greater than our significance level. So we are curious about why does it return 1, is it a perfect model ?

In addition to these, here's the output of my Logistic Regression Model

Logistic Regression Model

lrm(formula = bool.revenue.all.time ~ level + building.count +
gold.spent + npc + friends + post.count, data = sn, x = TRUE,
y = TRUE)

Model Likelihood     Discrimination    Rank Discrim.
Ratio Test            Indexes          Indexes
Obs         1e+05    LR chi2    1488.63    R2       0.147    C       0.774
0          99065    d.f.             6    g        1.141    Dxy     0.547
1            935    Pr(> chi2) <0.0001    gr       3.130    gamma   0.586
max |deriv| 8e-09                          gp       0.011    tau-a   0.010
Brier    0.009

Coef    S.E.   Wald Z Pr(>|Z|)
Intercept      -6.7910 0.0938 -72.36 <0.0001
level           0.0756 0.0193   3.92 <0.0001
building.count  0.0698 0.0091   7.64 <0.0001
gold.spent      0.0020 0.0002  11.05 <0.0001
npc             0.0172 0.0057   3.03 0.0024
friends         0.0304 0.0045   6.82 <0.0001
post.count     -0.0132 0.0042  -3.17 0.0015


This is validation with bootstrap's output

  index.orig training   test optimism index.corrected    n
Dxy           0.5511   0.5500 0.5506  -0.0006          0.5518 1000
R2            0.1469   0.1469 0.1465   0.0005          0.1465 1000
Intercept     0.0000   0.0000 0.0002  -0.0002          0.0002 1000
Slope         1.0000   1.0000 0.9997   0.0003          0.9997 1000
Emax          0.0000   0.0000 0.0001   0.0001          0.0001 1000
D             0.0149   0.0149 0.0148   0.0000          0.0148 1000
U             0.0000   0.0000 0.0000   0.0000          0.0000 1000
Q             0.0149   0.0149 0.0148   0.0001          0.0148 1000
B             0.0086   0.0086 0.0086   0.0000          0.0086 1000
g             1.1410   1.1381 1.1365   0.0016          1.1394 1000
gp            0.0111   0.0111 0.0111   0.0000          0.0111 1000


And this is the output of my calibration curve:

n=100000   Mean absolute error=0.002   Mean squared error=5e-05
0.9 Quantile of absolute error=0.002


Thanks.

• What do you mean by "accuracy"? Correctness of prediction? Sensitivity? Specificity? Something else? Jul 23, 2013 at 10:54
• Hi, I mean, Correctness of prediction. To understand how reliable it is. Jul 23, 2013 at 11:00
• That test does not have a $\chi^2$ distribution but rather a degenerate distribution because the degrees of freedom increases with $n$. To better understand accuracy, look at measures of calibration and predictive accuracy. See stats.stackexchange.com/questions/64788/… for a way to obtain multiple interesting estimates. Jul 23, 2013 at 11:23
• The $c$-index does not address calibration at all. The calibration curve certainly looks odd. What are the 100th lowest and highest predictions? Sometimes we need to not focus on the part of the calibration curve not supported by a sufficient sample size. On the other hand you could have a misspecified model. Jul 23, 2013 at 15:15
• f <- lrm(); n <- sum(f$freq); quantile(predict(f, type='fitted'), c(100/n, 1-100/n), na.rm=TRUE) Jul 23, 2013 at 15:23 ## 2 Answers If you want to assess accuracy, one way is to look at the predicted outcome vs. the actual outcome. You can get the predicted values with fitted-values and then compare them to the actual values; for one example see this page: • What is the difference between comparing the fitted vs. actual values of a logistic regression and calculating the predicted probabilities on a training data set and using them to test the predictive accuracy on a testing data set? – coip Feb 16, 2018 at 0:00 Try the below code: currMod <- glm(as.formula(form),family=binomial(link='logit'), data=inputData_signif_training) # build the model predictedval <- predict(currMod,newdata=inputData_signif_test[1:6],type='response') fitted.results.cat <- ifelse(predictedval > 0.5,"Yes","No") fitted.results.cat<-as.factor(fitted.results.cat) require(caret) cm<-confusionMatrix(data=fitted.results.cat, reference=inputData_signif_test$Heart.Failure)

Accuracy<-round(cm\$overall[1],2)

• Those are improper accuracy scoring rules requiring highly arbitrary binning of continuous probabilities. This problem was already solved using high-resolution calibration curves. Mar 22, 2016 at 12:14