27
$\begingroup$

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and should not be used to evaluate the quality of predictions generated by a probabilistic model such as logistic regression.

However, quite a few academic papers I have read have given misclassification loss as an example of a (non-strict) proper scoring rule in a binary classification setting. The clearest explanation I could find was in this paper, at bottom of page 7. To the best of my understanding, minimizing misclassification loss is equivalent to maximizing accuracy, and the equations in the paper make sense intuitively.

For example: using the notation of the paper, if the true conditional probability (given some feature vector x) of the class of interest is η = 0.7, any forecast q > 0.5 would have an an expected loss R(η|q) = 0.7(0) + 0.3(1) = 0.3, and any q $\leq$ 0.5 would have an expected loss of 0.7. The loss function would therefore be minimized at q = η = 0.7 and consequently proper; the generalization to the entire range of true conditional probabilities and forecasts seems straightforward enough from there.

Assuming the above calculations and statements are correct, the drawbacks of a non-unique minimum and all predictions above 0.5 sharing the same minimum expected loss are obvious. I still see no reason to use accuracy over the traditional alternatives such as log score, Brier score, etc. However, is it correct to say that accuracy is a proper scoring rule when evaluating probabilistic models in a binary setting, or am I making a mistake - either in my understanding of misclassification loss, or in equating it with accuracy?

$\endgroup$

1 Answer 1

31
$\begingroup$

TL;DR

Accuracy is an improper scoring rule. Don't use it.

The slightly longer version

Actually, accuracy is not even a scoring rule. So asking whether it is (strictly) proper is a category error. The most we can say is that under additional assumptions, accuracy is consistent with a scoring rule that is improper, discontinuous and misleading. (Don't use it.)

Your confusion

Your confusion stems from the fact that misclassification loss as per the paper you cite is not a scoring rule, either.

The details: scoring rules vs. classification evaluations

Let us fix terminology. We are interested in a binary outcome $y\in\{0,1\}$, and we have a probabilistic prediction $\widehat{q} = \widehat{P}(Y=1)\in(0,1)$. We know that $P(Y=1)=\eta>0.5$, but our model $\widehat{q}$ may or may not know that.

A scoring rule is a mapping that takes a probabilistic prediction $\widehat{q}$ and an outcome $y$ to a loss,

$$ s\colon (\widehat{q},y) \mapsto s(\widehat{q},y). $$

$s$ is proper if it is optimized in expectation by $\widehat{q}=\eta$. ("Optimized" usually means "minimized", but some authors flip signs and try to maximize a scoring rule.) $s$ is strictly proper if it is optimized in expectation only by $\widehat{q}=\eta$.

We will typically evaluate $s$ on many predictions $\widehat{q}_i$ and corresponding outcomes $y_i$ and average to estimate this expectation.

Now, what is accuracy? Accuracy does not take a probabilistic prediction as an argument. It takes a classification $\widehat{y}\in\{0,1\}$ and an outcome:

$$ a\colon (\widehat{y},y)\mapsto a(\widehat{y},y) = \begin{cases} 1, & \widehat{y}=y \\ 0, & \widehat{y} \neq y. \end{cases} $$

Therefore, accuracy is not a scoring rule. It is a classification evaluation. (This is a term I just invented; don't go looking for it in the literature.)

Now, of course we can take a probabilistic prediction like our $\widehat{q}$ and turn it into a classification $\widehat{y}$. But to do so, we will need the additional assumptions alluded to above. For instance, it is very common to use a threshold $\theta$ and classify:

$$ \widehat{y}(\widehat{q},\theta) := \begin{cases} 1, & \widehat{q}\geq \theta \\ 0, & \widehat{q}<\theta. \end{cases} $$

A very common threshold value is $\theta=0.5$. Note that if we use this threshold and then evaluate the accuracy over many predictions $\widehat{q}_i$ (as above) and corresponding outcomes $y_i$, then we arrive exactly at the misclassification loss as per Buja et al. Thus, misclassification loss is also not a scoring rule, but a classification evaluation.

If we take a classification algorithm like the one above, we can turn a classification evaluation into a scoring rule. The point is that we need the additional assumptions of the classifier. And that accuracy or misclassification loss or whatever other classification evaluation we choose may then depend less on the probabilistic prediction $\widehat{q}$ and more on the way we turn $\widehat{q}$ into a classification $\widehat{y}=\widehat{y}(\widehat{q},\theta)$. So optimizing the classification evaluation may be chasing after a red herring if we are really interested in evaluating $\widehat{q}$.

Now, what is improper about these scoring-rules-under-additional-assumptions? Nothing, in the present case. $\widehat{q}=\eta$, under the implicit $\theta =0.5$, will maximize accuracy and minimize misclassification loss over all possible $\widehat{q}\in(0,1)$. So in this case, our scoring-rule-under-additional-assumptions is proper.

Note that what is important for accuracy or misclassification loss is only one question: do we classify ($\widehat{y}$) everything as the majority class or not? If we do so, accuracy or misclassification loss are happy. If not, they aren't. What is important about this question is that it has only a very tenuous connection to the quality of $\widehat{q}$.

Consequently, our scoring-rules-under-additional-assumptions are not strictly proper, as any $\widehat{q}\geq\theta$ will lead to the same classification evaluation. We might use the standard $\theta=0.5$, believe that the majority class occurs with $\widehat{q}=0.99$ and classify everything as the majority class, because $\widehat{q}\geq\theta$. Accuracy is high, but we have no incentive to improve our $\widehat{q}$ to the correct value of $\eta$.

Or we might have done an extensive analysis of the asymmetric costs of misclassification and decided that the best classification probability threshold should actually be $\theta =0.2$. For instance, this could happen if $y=1$ means that you suffer from some disease. It might be better to treat you even if you don't suffer from the disease ($y=0$), rather than the other way around, so it might make sense to treat people even if there is a low predicted probability (small $\widehat{q}$) they suffer from it. We might then have a horrendously wrong model that believes that the true majority class only occurs with $\widehat{q}=0.25$ - but because of the costs of misclassification, we still classify everything as this (assumed) minority class, because again $\widehat{q}\geq\theta$. If we did this, accuracy or misclassification loss would make us believe we are doing everything right, even if our predictive model does not even get which one of our two classes is the majority one.

Therefore, accuracy or misclassification loss can be misleading.

In addition, accuracy and misclassification loss are improper under the additional assumptions in more complex situations where the outcomes are not iid. Frank Harrell, in his blog post Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules cites an example from one of his books where using accuracy or misclassification loss will lead to a misspecified model, since they are not optimized by the correct conditional predictive probability.

Another problem with accuracy and misclassification loss is that they are discontinuous as a function of the threshold $\theta$. Frank Harrell goes into this, too.

More information can be found at Why is accuracy not the best measure for assessing classification models?.

The bottom line

Don't use accuracy. Nor misclassification loss.

The nitpick: "strict" vs. "strictly"

Should we be talking about "strict" proper scoring rules, or about "strictly" proper scoring rules? "Strict" modifies "proper", not "scoring rule". (There are "proper scoring rules" and "strictly proper scoring rules", but no "strict scoring rules".) As such, "strictly" should be an adverb, not an adjective, and "strictly" should be used. As is more common in the literature, e.g., the papers by Tilmann Gneiting.

$\endgroup$
10
  • $\begingroup$ There are many aspects of your post I do not follow (or feel are not relevant to the question I asked), but let's start with "the misclassification loss as per the paper you cite is not a scoring rule." The formula is given very clearly in the paper: L1(1-q) = 1[q<=0.5] (pardon the poor formatting). It is, for all practical purposes, a step function that directly maps any probabilistic prediction and its associated outcome to a loss of 0 or 1. Furthermore, 0.5 is just a parameter that controls where the step occurs; I fail to see the "assumption" involved. How is this not a scoring rule? $\endgroup$
    – Zyzzva
    Jul 31, 2018 at 23:23
  • 2
    $\begingroup$ The threshold of 0.5 is the assumption. The probabilistic prediction $q$ is mapped to a classification by using the threshold, and the misclassification loss is then only a function of this classification. You could calculate the misclassification loss equally for any other classification, e.g., one that rolls a die and assigns an instance to class A if we roll a 1 or 2. I did my best to explain what is a complicated and often misunderstood topic (and I do feel that everything I write about is relevant); I am sorry if I did not succeed. I would be happy to discuss any remaining points. $\endgroup$ Aug 1, 2018 at 6:13
  • 1
    $\begingroup$ As for the relevancy comment, I apologize if it came off the wrong way. I tried to focus the scope of the question to be specifically about proper vs. improper, not discontinuous/misleading/etc. I am well acquainted with the links you provided and have no issues with your comments on misclassification costs or bottom line. I am just seeking a more rigorous explanation of the statement "accuracy is improper", especially given that this paper suggests otherwise for the common use case of binary outcomes. I appreciate you taking the time to discuss this with me and share your detailed thoughts. $\endgroup$
    – Zyzzva
    Aug 1, 2018 at 9:47
  • 1
    $\begingroup$ After further reflection, I think I have a clearer grasp of the point you are making. If we consider the same step function with the step at 0.6 (corresponding to classification at a threshold of 0.6), then the scoring rule is improper, because expected loss will no longer be minimized by a prediction q = n for n in the range [0.5, 0.6]. More generally, it will be improper at every threshold other than 0.5, and often in practice we want to use other thresholds due to asymmetric costs of misclassification, as you pointed out. $\endgroup$
    – Zyzzva
    Aug 1, 2018 at 12:32
  • 1
    $\begingroup$ I concur that accuracy is clearly a bad metric for evaluating probabilities, even when a threshold of 0.5 is justified. I did say as much at the end of the original post I made, but this helped clear up the specific details I was having trouble with - namely, reconciling something I misunderstood as showing that accuracy is proper for binary outcomes (when it reality it only applies to the very specific case of a 0.5 threshold) with the seemingly black-and-white statement "accuracy is improper" that I have been seeing a lot. Thanks for your help and patience. $\endgroup$
    – Zyzzva
    Aug 1, 2018 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.