Taking the sum of predicted probabilities from logit model?

Question

I am using a logit model to predict the probability that students pass a particular course. I run the logit, generate predicted probabilities for the students in my sample, and want to compare the model with observed pass rates. I create classification tables and perform some other checks.

One approach that has been suggested to me is to sum all of the predicted probabilities for the sample (and within subgroups of interest) to compare the predicted number of passes to the observed number of passes. Intuitively, this makes sense to me. It seems reasonable that summing the predicted probabilities would produce the expected total number of events the model is predicting in my sample. However, as I try to find peer-reviewed justification for this, I am coming up short and unable to find verified examples of this being used.

My question is: Is it appropriate to sum the predicted probabilities for my sample and for subgroups of my sample to compare expected event counts to the observed event counts, and if so, is this method validated or used in any reputable sources?

Answer 1

No. To see why, consider the following example.

We have a class in which 50% of the students pass. Our model predicts that every student who passed had a 0% chance of passing, and every student that failed had a 100% chance of passing. If we sum the probabilities, we find that the model predicts that exactly 50% of the students pass, so is, informally, 100% correct.

Another model that predicts every student will pass with a probability of 50% will do as well.

A model that predicts the passing students have a 90% probability of passing while the failing students have a 20% chance of passing will perform worse than either of the two models above (it predicts a 55% pass rate) but is clearly the best of the three at an individual level.

Generally speaking, for this type of problem, we want to use a proper scoring rule , such as log-loss or squared error. Using squared error (i.e., the squared difference between a 0-1 fail/pass score and the probabilistic prediction), the third model above would be the best of the three.