What is a reliable measure of accuracy for logistic regression?

Question

Say I fit a logistic regression model on training data and test it on test data.

How do I measure accuracy?

1. Using the "treshold method" where we predict $y = 1$ if the predicted probability is above 0.5, and vice versa, is a very poor method, obviously. I think it would only work well for simple problems but not for complicated data.
2. A second method I know is to calculate a $\sum_{i}$ where each term is either $\log p_i$ if the actual value at the point $i$ is $1$, or $\log(1 - p_i)$ if the actual value is $0$. But testing this on my data, I do not get sensible results. This method seems to be biased in preference of models that predict $p_i \approx 50 \%$ rather than predict extreme probabilities....

What is a simple and robust method?

Answer 1

1. Thresholding and assessing Accuracy is indeed not a good idea. See, for instance:

   • Why is accuracy not the best measure for assessing classification models?
   • Classification probability threshold

2. What you are describing is known as the "log score", which is a proper scoring rule. It will be maximized (in expectation) by any method that outputs correct probabilistic predictions $p_i$.

I would always recommend using the log score, or any other proper scoring rule. If you "do not get sensible results", that may be worth a separate question on CV.

It may simply be that your classes are not well separated. If two instances have the exact same attributes, but will belong to the different classes with roughly equal probability, then the best any method can do is predict $\hat{p}\approx 0.5$ for both instances. You can't expect predicted probabilities that are close to 0 or 1; in particular, you can't expect proabilities that differ (since we are assuming that both instances have the same attributes). In such a situation, the true $p$ is indeed about 0.5, so $\hat{p}\approx 0.5$ is indeed a good prediction. It simply tells you that there is little to know for these particular instances. See the first link above.