# Evaluating a binomial (success vs. failure) glm

I'm familiar with (some) approaches to evaluating the fit (or accuracy) of a binary (logistic) model (e.g. AUC). Are there methods/approaches that are particularly well-suited for a binomial (success vs. failure) model?

If the suggestion is to use a variant of a (pseudo-) R-square, what are recommended approaches? I can think of a few, using logit vs. response-scale predictions and with and without weighting by # of subjects, but am unsure of their appropriateness:

[s = # of successes, f = # of failures]

1. summary(lm(predict(model)~I(s/(s+f))))$r.square 2. summary(lm(predict(model,type='response')~I(s/(s+f))))$r.square

3. summary(lm(predict(model)~I(s/(s+f)),weights=s+f))$r.square 4. summary(lm(predict(model,type='response')~I(s/(s+f)),weights=s+f))$r.square

5. 1-var(residuals(model))/(var(s/(s+f)))

• A binary response is just a special case of a binomial response, with the binomial denominator being equal to 1. So all the goodness-of-fit measures for a binary model will work just as well in the binomial case. – Hong Ooi Oct 16 '14 at 6:59
• Thanks. As both an "accuracy statistic" like AUC and a g-o-f test like Hosmer-Lemeshow that are typically applied in a binary (logistic) model require a binary response, it is unclear to me how these (or others) would work with a proportional response. – Giancarlo Oct 16 '14 at 7:13
• A proportional response $S/(S+F)$ is just a combination of $S$ successes and $F$ failures. You might have to do some arithmetic, but it should be clear from that how to extend the formulas. – Hong Ooi Oct 16 '14 at 7:35
• Perhaps less clear than you might think. Let's take one observation as an example to see if I follow you: while a test of a binary model might compare an observed value of 0 and a model-predicted value of 0.3, could be this extended to a observed proportion of 0.25 (1 S vs. 3 F) and predicted value of 0.3 by comparing two vectors of 1,0,0,0 and 0.3,0.3,0.3,0.3? – Giancarlo Oct 16 '14 at 16:58

In regression, a binomial response is basically a compact way of representing multiple (independent) binary observations that have the same values of the predictors. From that, you can decompose a single observation with the proportion $S/(S + F)$ into $S + F$ observations: $S$ successes and $F$ failures. Note that you do need to know both the numerator and denominator of the proportion; you can't get by with just the proportion itself.
To take your example of $S = 1$ and $F = 3$, and a predicted probability of $0.3$: you would treat this as 1 case with a binary response value of $1$, and 3 cases with a binary response of $0$. So yes, you are comparing the two vectors $Y_\text{obs} = \{1,0,0,0\}$ and $\hat{Y} = \{0.3, 0.3, 0.3, 0.3\}$.