7
$\begingroup$

Is there a standard way to report the percent correctly predicted when predicting a binary outcome? Using glm in r, the results are predicted probabilities. However, in order to make a comparison to another model, I want to report a single percent correctly predicted value from my binary model. Do I simply choose a cutpoint, and if so, how? Here is a simple example of the code.

model.results <- glm(binary.outcome ~ predictor1 + predictor2, family=quasibinomial)

Thanks,

$\endgroup$
3
  • 2
    $\begingroup$ That is an improper accuracy scoring rule so will give misleading model comparisons. $\endgroup$ Nov 29, 2012 at 19:10
  • $\begingroup$ Isn't this a duplicate of Classification table for logistic regression in R? $\endgroup$
    – chl
    Nov 29, 2012 at 21:01
  • $\begingroup$ I don't think so -- that question uses .5 as the break point -- I'm asking what the appropriate breakpoint is. $\endgroup$
    – mike
    Nov 29, 2012 at 21:26

1 Answer 1

8
$\begingroup$

@FrankHarrell is correct that percent accuracy isn't the loss function that logistic regression is trying to optimize. So there could be situations where the best model according to the (quasi) binomial likelihood isn't also the best one according to percent accuracy.

Edited to add: He's also right in the comments below that setting a cutpoint has serious problems. What I've proposed below is a workaround that gets at the intuition of percent accuracy but avoids setting an arbitrary threshold between the underlying continuous model predictions.

On the other hand, percent accuracy seems like a perfectly reasonable loss function as well, and it might be worth knowing how logistic regression performs with it. Percent accuracy can be more intuitive, and it isn't as susceptible to outliers and the occasional prediction that was off by a very large amount.

Finding this value is pretty straightforward. First, find the probabilities the model assigns to each outcome:

probabilities <- predict(model.results, type = "response")

If your glm flipped a bunch of biased coins for each response, it would give heads this percent of the time. Then all you need to do is find the proportion of the time that the coin will come up the wrong way. The simplest way is probably:

1 - mean(abs(probabilities - binary.outcome))

You can prove to yourself that this gives the right answer by simulating the biased coinflips yourself with rbinom.

$\endgroup$
6
  • 1
    $\begingroup$ The fact that it is intuitive is not a reason to use it. Oversimplified concepts are often intuitive. It is not a reasonable loss function unless the ultimate user has a discontinuous loss function with a utility function that happens to coincide with the utility function implied by the analyst's choice of cutpoint. That's a lot to ask. Improper scoring rules will result in the wrong model being chosen, by many other reasonable measures, not just the log likelihood. $\endgroup$ Nov 30, 2012 at 15:30
  • $\begingroup$ P.S. Another way the problem with classification proportion shows itself is that you get different models with different cutoffs on what is a predicted "positive" and no two people may be able to agree on which cutoff is "correct". $\endgroup$ Nov 30, 2012 at 17:06
  • $\begingroup$ What cutpoint are you referring to? This is just the L1 distance between the predicted probability and what was observed. I suppose its derivative is discontinuous, but I'm not sure why that's a problem. $\endgroup$ Nov 30, 2012 at 21:58
  • $\begingroup$ To clarify, this is the expected accuracy if you flipped coins from the model's predictions, not the accuracy you'd get if you set an arbitrary cutoff. $\endgroup$ Nov 30, 2012 at 22:01
  • 2
    $\begingroup$ Sorry I had misinterpreted that part of your note. The mean absolute error is reasonable and has much written about it in this context. It is almost the same as the ordinary $R^2$ because of special properties of squaring things that are Y=0 and Y=1. See Buyse, M. $R^2$: A useful measure of model performance when predicting a dichotomous outcome Statistics in Medicine, 2000, 19, 271-274 and Hu, B.; Palta, M. & Shao, J. Properties of $R^2$ statistics for logistic regression Statistics in Medicine, 2006, 25, 1383-1395 $\endgroup$ Nov 30, 2012 at 23:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.