How to decide the "best" accuracy score for prediction of binary outcome?

Question

Dr Frank Harrell mentioned in his book and BIOS 330 course that

Accuracy score used to drive model building should be a continuous score that utilizes all the information in the data (e.g. Brier score, log likelihood, deviance, mean square error)

I am wondering:

1. In what sense are these scores "continuous"? Is it continuous when we view it as a mapping from a topological space (input dataset) to $\mathbb{R}$? What would then be the topology/metric on the event space?
2. How is Brier score better than "the proportion classified correctly" as an accuracy score, as Brier score is also sensitive to the relative frequencies of the outcome variable? Consider a non-informative model of always predicting 1 with probability 1, the Brier score would be very different if the true prevalence is 0.30 or 0.005. Or maybe I am not understanding the sensitivity here correctly.
3. How do we choose among the continuous scores? We have Brier score, log likelihood, deviance for the binary prediction case. How do we decide which one would give us the "best" model?

Answer 1

1. The scores are continuous not (necessarily) in the sense of small perturbations to the input data, but rather in small perturbations of the prediction model. Of course, small perturbations in input data will often yield small perturbations in the model.

   If you have probabilistic predictions for a discrete classification and perturb these probabilities slightly, scores will only change slightly.

   In contrast, assume you output non-probabilistic classifications that are based on these probabilities and a probability threshold, and then assess quality via accuracy, precision or similar. If you perturb the probabilities or the threshold slightly, then the classifications will not change, nor will accuracy/precision. However, at slightly larger perturbations, the first cases will discretely change classification, and at this point, accuracy/precision will change by a discrete step.

2. Yes, scores will depend on the underlying prevalence. But this is typically considered as given, whereas the thing we want to vary is the predictive model, so this is not a problem. (And in any case, as the prevalence changes, scores will change continuously with it.)

3. How to choose among different possible scoring rules is a thornier problem. Merkle & Steyvers (2013, Decision Analysis) point out that the Brier and the logarithmic scores are members of a two-parameter family of proper scoring rules (of course, not all members are strictly proper). They give a few guidelines on how to choose a rule and point out that "Researchers often find that one’s choice of strictly proper scoring rule has minimal impact on one’s conclusions", at least if we restrict ourselves to the "classical" scoring rules.