I'm familiar with (some) approaches to evaluating the fit (or accuracy) of a binary (logistic) model (e.g. AUC). Are there methods/approaches that are particularly well-suited for a binomial (success vs. failure) model?

If the suggestion is to use a variant of a (pseudo-) R-square, what are recommended approaches? I can think of a few, using logit vs. response-scale predictions and with and without weighting by # of subjects, but am unsure of their appropriateness:

[s = # of successes, f = # of failures]

  1. summary(lm(predict(model)~I(s/(s+f))))$r.square

  2. summary(lm(predict(model,type='response')~I(s/(s+f))))$r.square

  3. summary(lm(predict(model)~I(s/(s+f)),weights=s+f))$r.square

  4. summary(lm(predict(model,type='response')~I(s/(s+f)),weights=s+f))$r.square

  5. 1-var(residuals(model))/(var(s/(s+f)))

  • 1
    $\begingroup$ A binary response is just a special case of a binomial response, with the binomial denominator being equal to 1. So all the goodness-of-fit measures for a binary model will work just as well in the binomial case. $\endgroup$ – Hong Ooi Oct 16 '14 at 6:59
  • $\begingroup$ Thanks. As both an "accuracy statistic" like AUC and a g-o-f test like Hosmer-Lemeshow that are typically applied in a binary (logistic) model require a binary response, it is unclear to me how these (or others) would work with a proportional response. $\endgroup$ – Giancarlo Oct 16 '14 at 7:13
  • 1
    $\begingroup$ A proportional response $S/(S+F)$ is just a combination of $S$ successes and $F$ failures. You might have to do some arithmetic, but it should be clear from that how to extend the formulas. $\endgroup$ – Hong Ooi Oct 16 '14 at 7:35
  • $\begingroup$ Perhaps less clear than you might think. Let's take one observation as an example to see if I follow you: while a test of a binary model might compare an observed value of 0 and a model-predicted value of 0.3, could be this extended to a observed proportion of 0.25 (1 S vs. 3 F) and predicted value of 0.3 by comparing two vectors of 1,0,0,0 and 0.3,0.3,0.3,0.3? $\endgroup$ – Giancarlo Oct 16 '14 at 16:58

In regression, a binomial response is basically a compact way of representing multiple (independent) binary observations that have the same values of the predictors. From that, you can decompose a single observation with the proportion $S/(S + F)$ into $S + F$ observations: $S$ successes and $F$ failures. Note that you do need to know both the numerator and denominator of the proportion; you can't get by with just the proportion itself.

To take your example of $S = 1$ and $F = 3$, and a predicted probability of $0.3$: you would treat this as 1 case with a binary response value of $1$, and 3 cases with a binary response of $0$. So yes, you are comparing the two vectors $Y_\text{obs} = \{1,0,0,0\}$ and $\hat{Y} = \{0.3, 0.3, 0.3, 0.3\}$.

  • $\begingroup$ Thanks for clarifying/confirming (I'd upvote, but don't have enough points yet). You don't happen to know of any examples in the literature employing this approach? As an aside, I saw a similar (unanswered) question on StackOverflow stackoverflow.com/questions/22669098/… $\endgroup$ – Giancarlo Oct 17 '14 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.