# Binary outcomes with low goodness of fit (GOF) but good predictive power?

We know that it is possible that a model predicts the outcome reasonably well (and hence high R-square) but is actually misspecified (and hence low goodness of fit), as this example shows.

The outcome of that example is a continuous variable. I am sure there are thousands of similar examples for binary outcomes, but it is just hard for me to imagine one. Can anyone find such an example for me that shows a model predicts the binary outcomes well but is actually misspecified, causing low goodness of fit? I'd appreciate it.

## 1 Answer

You might want to have a look at Esarey and Pearce (2012) Assessing Fit Quality and Testing for Misspecification in Binary-Dependent Variable Models (a more recent version of this paper has been published but I don't have the reference on me and not sure if it's not behind a paywall). Measuring goodness of fit is also discussed in a full chapter in Hosmer, Lemeshow and Sturdivant.

My reading is that it partly depends on the method you use to assess goodness of fit. With binary logistic regression, we're ultimately interested in the probability that y=1 or y=0. Esarey and Pearce show that you can either have a mis-specified model that appears to behave well, or a well-specified model that appears to behave badly.

For example, using the area under the receiver operator characteristic curve, which is used to asses a model's discriminative ability (how well does it classify y=1 and y=0), you can get a 'poor' result because this particular test is actually assessing how well the model separates 1s from 0s, not how well it tracks probabilites. In addition, the AUC relies on setting an arbitrary cut-point (say, 0.5) on when we say the model has predicted a 1 or 0. This may be entirely inappropriate if the outcome is rare because no unit will have a probability of >=0.5. There is a concrete example on p 6 et seq of the above article.