170
$\begingroup$

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.

Accuracy, the proportion of correct classifications among all classifications, is very simple and very "intuitive" measure, yet it may be a poor measure for imbalanced data. Why does our intuition misguide us here and are there any other problems with this measure?

$\endgroup$
0

10 Answers 10

168
+50
$\begingroup$

Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes.

Frank Harrell has written about this on his blog: Classification vs. Prediction and Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules.

Essentially, his argument is that the statistical component of your exercise ends when you output a probability for each class of your new sample. Mapping these predicted probabilities $(\hat{p}, 1-\hat{p})$ to a 0-1 classification, by choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component. And here, you need the probabilistic output of your model - but also considerations like:

  • What are the consequences of deciding to treat a new observation as class 1 vs. 0? Do I then send out a cheap marketing mail to all 1s? Or do I apply an invasive cancer treatment with big side effects?
  • What are the consequences of treating a "true" 0 as 1, and vice versa? Will I tick off a customer? Subject someone to unnecessary medical treatment?
  • Are my "classes" truly discrete? Or is there actually a continuum (e.g., blood pressure), where clinical thresholds are in reality just cognitive shortcuts? If so, how far beyond a threshold is the case I'm "classifying" right now?
  • Or does a low-but-positive probability to be class 1 actually mean "get more data", "run another test"?

Depending on the consequences of your decision, you will use a different threshold to make the decision. If the action is invasive surgery, you will require a much higher probability for your classification of the patient as suffering from something than if the action is to recommend two aspirin. Or you might even have three different decisions although there are only two classes (sick vs. healthy): "go home and don't worry" vs. "run another test because the one we have is inconclusive" vs. "operate immediately".

The correct way of assessing predicted probabilities $(\hat{p}, 1-\hat{p})$ is not to compare them to a threshold, map them to $(0,1)$ based on the threshold and then assess the transformed $(0,1)$ classification. Instead, one should use proper . These are loss functions that map predicted probabilities and corresponding observed outcomes to loss values, which are minimized in expectation by the true probabilities $(p,1-p)$. The idea is that we take the average over the scoring rule evaluated on multiple (best: many) observed outcomes and the corresponding predicted class membership probabilities, as an estimate of the expectation of the scoring rule.

Note that "proper" here has a precisely defined meaning - there are improper scoring rules as well as proper scoring rules and finally strictly proper scoring rules. Scoring rules as such are loss functions of predictive densities and outcomes. Proper scoring rules are scoring rules that are minimized in expectation if the predictive density is the true density. Strictly proper scoring rules are scoring rules that are only minimized in expectation if the predictive density is the true density.

As Frank Harrell notes, accuracy is an improper scoring rule. (More precisely, accuracy is not even a scoring rule at all: see my answer to Is accuracy an improper scoring rule in a binary classification setting?) This can be seen, e.g., if we have no predictors at all and just a flip of an unfair coin with probabilities $(0.6,0.4)$. Accuracy is maximized if we classify everything as the first class and completely ignore the 40% probability that any outcome might be in the second class. (Here we see that accuracy is problematic even for balanced classes.) Proper will prefer a $(0.6,0.4)$ prediction to the $(1,0)$ one in expectation. In particular, accuracy is discontinuous in the threshold: moving the threshold a tiny little bit may make one (or multiple) predictions change classes and change the entire accuracy by a discrete amount. This makes little sense.

More information can be found at Frank's two blog posts linked to above, as well as in Chapter 10 of Frank Harrell's Regression Modeling Strategies.

(This is shamelessly cribbed from an earlier answer of mine.)


EDIT. My answer to Example when using accuracy as an outcome measure will lead to a wrong conclusion gives a hopefully illustrative example where maximizing accuracy can lead to wrong decisions even for balanced classes.

$\endgroup$
42
  • 8
    $\begingroup$ @Tim Frank's point (that he discussed in numerous answers on our site and elsewhere), as I understand it, is that if a classification algorithm does not return probabilities then it's garbage and should not be used. To be honest, most of the commonly used algorithms do return probabilities. $\endgroup$
    – amoeba
    Nov 9 '17 at 9:17
  • 12
    $\begingroup$ I'd say that an algorithm that takes past observations and outputs only classifications without taking the points above into account (e.g., costs of mis-decisions) conflates the statistical and the decision aspect. It's like someone recommending a particular type of car to you without first asking you whether you want to transport a little league baseball team, a bunch of building materials, or only yourself. So I'd also say such an algorithm would be garbage. $\endgroup$ Nov 9 '17 at 9:23
  • 12
    $\begingroup$ I was going to write an answer, but then didn't need to. Bravo. I discuss this with my students as a "separation of concerns" between statistical modeling and decision making. This type of concept is very deeply rooted in engineering culture. $\endgroup$ Nov 9 '17 at 14:34
  • 9
    $\begingroup$ @chainD: if your classifier (remember, it's the one with the highest accuracy) says that "everyone in this sample is healthy", then what doctor or analyst would believe that there is more to the story? I agree that in the end, it's a call for the analyst to make, but "everyone is healthy" is far less helpful to the analyst than something that draws attention to residual uncertainty like the 95%/5% prediction. $\endgroup$ Nov 10 '17 at 20:43
  • 15
    $\begingroup$ @StephanKolassa 's answer and comments are superb. Someone else comment implied that there is a difference in how this is viewed depending on which culture you are part of. This is not really the case; it's just that some fields bothered to understand the literature and others didn't. Weather forecasting, for example, has been at the forefront and has used proper scoring rules for assessing forecaster accuracy since at least 1951. $\endgroup$ Nov 11 '17 at 12:53
96
$\begingroup$

When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a great way to lower the cost. Predict that every instance belongs to the majority class, get accuracy of 99% and go home early.

The problem starts when the actual costs that we assign to every error are not equal. If we deal with a rare but fatal disease, the cost of failing to diagnose the disease of a sick person is much higher than the cost of sending a healthy person to more tests.

In general, there is no general best measure. The best measure is derived from your needs. In a sense, it is not a machine learning question, but a business question. It is common that two people will use the same data set but will choose different metrics due to different goals.

Accuracy is a great metric. Actually, most metrics are great and I like to evaluate many metrics. However, at some point you will need to decide between using model A or B. There you should use a single metric that best fits your need.

For extra credit, choose this metric before the analysis, so you won't be distracted when making the decision.

$\endgroup$
9
  • 4
    $\begingroup$ Great answer - I've proposed a couple of edits just to try and make the point clearer to beginners in machine learning (at whom this question is aimed). $\endgroup$
    – nekomatic
    Nov 9 '17 at 8:47
  • 1
    $\begingroup$ I'd disagree that it's not a machine learning problem. But addressing it would involve doing machine learning on the meta problem and necessitate the machine having access to some kind of data beyond just the basic classification information. $\endgroup$ Nov 9 '17 at 20:41
  • 3
    $\begingroup$ I don't see it as a function of only the data since different goals can lad to different cost/model/performance/metrics. I do agree that in general, the question of cost can be handled mathematically. However questions like the cost of treating patients rely on totally different information. This information needed for the meta data is usually not suitable for machine learning methodology so most of the time it is handled with different methods. $\endgroup$
    – DaL
    Nov 12 '17 at 12:01
  • 2
    $\begingroup$ By "misdiagnosing a person with the disease", you mean "misdiagnosing a person who has the disease (as not having the disease)", right? Because that phrase could be interpreted either way. $\endgroup$ Nov 13 '17 at 1:39
  • 1
    $\begingroup$ I agree that they are useful. Please note that it is very easy to trade them off (e.g., by the confidence threshold) so you indeed use them together or use a combined measure (e.g., Jaccard. f scores). $\endgroup$
    – DaL
    Dec 20 '17 at 15:10
26
$\begingroup$

The problem with accuracy

Standard accuracy is defined as the ratio of correct classifications to the number of classifications done.

\begin{align*} accuracy := \frac{\text{correct classifications}}{\text{number of classifications}} \end{align*}

It is thus an overall measure over all classes and as we'll shortly see it's not a good measure to tell an oracle apart from an actual useful test. An oracle is a classification function that returns a random guess for each sample. Likewise, we want to be able to rate the classification performance of our classification function. Accuracy can be a useful measure if we have the same amount of samples per class but if we have an imbalanced set of samples accuracy isn't useful at all. Even more so, a test can have a high accuracy but actually perform worse than a test with a lower accuracy.

If we have a distribution of samples such that $90\%$ of samples belong to class $\mathcal{A}$, $5\%$ belonging to $\mathcal{B}$ and another $5\%$ belonging to $\mathcal{C}$ then the following classification function will have an accuracy of $0.9$:

\begin{align*} classify(sample) := \begin{cases} \mathcal{A} & \text{if }\top \\ \end{cases} \end{align*}

Yet, it is obvious given that we know how $classify$ works that this it can not tell the classes apart at all. Likewise, we can construct a classification function

\begin{align*} classify(sample) := \text{guess} \begin{cases} \mathcal{A} & \text{with p } = 0.96 \\ \mathcal{B} & \text{with p } = 0.02 \\ \mathcal{C} & \text{with p } = 0.02 \\ \end{cases} \end{align*}

which has an accuracy of $0.96 \cdot 0.9 + 0.02 \cdot 0.05 \cdot 2 = 0.866$ and will not always predict $\mathcal{A}$ but still given that we know how $classify$ works it is obvious that it can not tell classes apart. Accuracy in this case only tells us how good our classification function is at guessing. This means that accuracy is not a good measure to tell an oracle apart from a useful test.

Accuracy per Class

We can compute the accuracy individually per class by giving our classification function only samples from the same class and remember and count the number of correct classifications and incorrect classifications then compute $accuracy := \text{correct}/(\text{correct} + \text{incorrect})$. We repeat this for every class. If we have a classification function that can accurately recognize class $\mathcal{A}$ but will output a random guess for the other classes then this results in an accuracy of $1.00$ for $\mathcal{A}$ and an accuracy of $0.33$ for the other classes. This already provides us a much better way to judge the performance of our classification function. An oracle always guessing the same class will produce a per class accuracy of $1.00$ for that class, but $0.00$ for the other class. If our test is useful all the accuracies per class should be $>0.5$. Otherwise, our test isn't better than chance. However, accuracy per class does not take into account false positives. Even though our classification function has a $100\%$ accuracy for class $\mathcal{A}$ there will also be false positives for $\mathcal{A}$ (such as a $\mathcal{B}$ wrongly classified as a $\mathcal{A}$).

Sensitivity and Specificity

In medical tests sensitivity is defined as the ratio between people correctly identified as having the disease and the amount of people actually having the disease. Specificity is defined as the ratio between people correctly identified as healthy and the amount of people that are actually healthy. The amount of people actually having the disease is the amount of true positive test results plus the amount of false negative test results. The amount of actually healthy people is the amount of true negative test results plus the amount of false positive test results.

Binary Classification

In binary classification problems there are two classes $\mathcal{P}$ and $\mathcal{N}$. $T_{n}$ refers to the number of samples that were correctly identified as belonging to class $n$ and $F_{n}$ refers to the number of samples that werey falsely identified as belonging to class $n$. In this case sensitivity and specificity are defined as following:

\begin{align*} sensitivity := \frac{T_{\mathcal{P}}}{T_{\mathcal{P}}+F_{\mathcal{N}}} \\ specificity := \frac{T_{\mathcal{N}}}{T_{\mathcal{N}}+F_{\mathcal{P}}} \end{align*}

$T_{\mathcal{P}}$ being the true positives $F_{\mathcal{N}}$ being the false negatives, $T_{\mathcal{N}}$ being the true negatives and $F_{\mathcal{P}}$ being the false positives. However, thinking in terms of negatives and positives is fine for medical tests but in order to get a better intuition we should not think in terms of negatives and positives but in generic classes $\alpha$ and $\beta$. Then, we can say that the amount of samples correctly identified as belonging to $\alpha$ is $T_{\alpha}$ and the amount of samples that actually belong to $\alpha$ is $T_{\alpha} + F_{\beta}$. The amount of samples correctly identified as not belonging to $\alpha$ is $T_{\beta}$ and the amount of samples actually not belonging to $\alpha$ is $T_{\beta} + F_{\alpha}$. This gives us the sensitivity and specificity for $\alpha$ but we can also apply the same thing to the class $\beta$. The amount of samples correctly identified as belonging to $\beta$ is $T_{\beta}$ and the amount of samples actually belonging to $\beta$ is $T_{\beta} + F_{\alpha}$. The amount of samples correctly identified as not belonging to $\beta$ is $T_{\alpha}$ and the amount of samples actually not belonging to $\beta$ is $T_{\alpha} + F_{\beta}$. We thus get a sensitivity and specificity per class:

\begin{align*} sensitivity_{\alpha} := \frac{T_{\alpha}}{T_{\alpha}+F_{\beta}} \\ specificity_{\alpha} := \frac{T_{\beta}}{T_{\beta} + F_{\alpha}} \\ sensitivity_{\beta} := \frac{T_{\beta}}{T_{\beta}+F_{\alpha}} \\ specificity_{\beta} := \frac{T_{\alpha}}{T_{\alpha} + F_{\beta}} \\ \end{align*}

We however observe that $sensitivity_{\alpha} = specificity_{\beta}$ and $specificity_{\alpha} = sensitivity_{\beta}$. This means that if we only have two classes we don't need sensitivity and specificity per class.

N-Ary Classification

Sensitivity and specificity per class isn't useful if we only have two classes, but we can extend it to multiple classes. Sensitivity and specificity is defined as:

\begin{align*} \text{sensitivity} := \frac{\text{true positives}}{\text{true positives} + \text{false negatives}} \\ \text{specificity} := \frac{\text{true negatives}}{\text{true negatives} + \text{false-positives}} \\ \end{align*}

The true positives is simply $T_{n}$, the false negatives is simply $\sum_{i}(F_{n,i})$ and the false positives is simply $\sum_{i}(F_{i,n})$. Finding the true negatives is much harder but we can say that if we correctly classify something as belonging to a class different than $n$ it counts as a true negative. This means we have at least $\sum_{i}(T_{i}) - T(n)$ true negatives. However, this aren't all true negatives. All the wrong classifications for a class different than $n$ are also true negatives, because they correctly weren't identified as belonging to $n$. $\sum_{i}(\sum_{k}(F_{i,k}))$ represents all wrong classifications. From this we have to subtract the cases where the input class was $n$ meaning we have to subtract the false negatives for $n$ which is $\sum_{i}(F_{n,i})$ but we also have to subtract the false positives for $n$ because they are false positives and not true negatives so we have to also subtract $\sum_{i}(F_{i,n})$ finally getting $\sum_{i}(T_{i}) - T(n) + \sum_{i}(\sum_{k}(F_{n,i})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n})$. As a summary we have:

\begin{align*} \text{true positives} := T_{n} \\ \text{true negatives} := \sum_{i}(T_{i}) - T(n) + \sum_{i}(\sum_{k}(F_{n,i})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n}) \\ \text{false positives} := \sum_{i}(F_{i,n}) \\ \text{false negatives} := \sum_{i}(F_{n,i}) \end{align*}

\begin{align*} sensitivity(n) := \frac{T_{n}}{T_{n} + \sum_{i}(F_{n,i})} \\ specificity(n) := \frac{\sum_{i}(T_{i}) - T_{n} + \sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n})}{\sum_{i}(T_{i}) - T_{n} + \sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{n,i})} \end{align*}

Introducing Confidence

We define a $confidence^{\top}$ which is a measure of how confident we can be that the reply of our classification function is actually correct. $T_{n} + \sum_{i}(F_{i,n})$ are all cases where the classification function replied with $n$ but only $T_{n}$ of those are correct. We thus define

\begin{align*} confidence^{\top}(n) := \frac{T_{n}}{T_{n}+\sum_{i}(F_{i,n})} \end{align*}

But can we also define a $confidence^{\bot}$ which is a measure of how confident we can be that if our classification function responds with a class different than $n$ that it actually wasn't an $n$?

Well, we get $\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}$ all of which are correct except $\sum_{i}(F_{n,i})$.Thus, we define

\begin{align*} confidence^{\bot}(n) = \frac{\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}-\sum_{i}(F_{n,i})}{\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}} \end{align*}

$\endgroup$
3
  • $\begingroup$ Can you please provide any example of calculating Mean Accuracy using confusion matrix. $\endgroup$ Sep 27 '18 at 1:17
  • $\begingroup$ You can find a more detailed description with examples here: mroman.ch/guides/sensspec.html $\endgroup$
    – mroman
    Sep 27 '18 at 14:40
  • $\begingroup$ Reading through it again there's an error in the definition of confidence_false. I'm surprised nobody spotted that. I'll fix that in the next few days. $\endgroup$
    – mroman
    Sep 27 '18 at 16:14
14
$\begingroup$

Imbalanced classes in your dataset

To be short: imagine, 99% of one class (say apples) and 1% of another class is in your data set (say bananas). My super duper algorithm gets an astonishing 99% accuracy for this data set, check it out:

return "it's an apple"

He will be right 99% of the time and therefore gets a 99% accuracy. Can I sell you my algorithm?

Solution: don't use an absolute measure (accuracy) but a relative-to-each-class measure (there are a lot out there, like ROC AUC)

$\endgroup$
11
  • $\begingroup$ Nope, AUC is also not appropriate for imbalanced dataset. $\endgroup$
    – SiXUlm
    Aug 16 '19 at 21:39
  • $\begingroup$ @SiXUlm, can you elaborate on that? $\endgroup$
    – Mayou36
    Aug 17 '19 at 0:01
  • $\begingroup$ AUC is the area under ROC curve. The ROC curve is the plot of TPR vs FPR. Now, in the Bayesian setting, the imbalance is the odd of prior probability: $P(D)/P(D^C)$. The TPR can be seen as $P(T \vert D)$ and FPR can be seen as $P(F \vert D^C)$. The prior probability has nothing to do with the likelihood. $\endgroup$
    – SiXUlm
    Aug 17 '19 at 23:38
  • $\begingroup$ A clearer illustration can be found here: quora.com/…. Have a look at Jerry Ma's answer. $\endgroup$
    – SiXUlm
    Aug 17 '19 at 23:42
  • 5
    $\begingroup$ AUC is actually appropriate for an imbalanced dataset, in fact it is one of the better metrics. In the extreme case; it is a relatively insensitive to false positives in the positive fraction compared with precision recall curves, but saying it is inappropiate for class imbalance is just incorrect. $\endgroup$ Oct 3 '19 at 12:51
5
$\begingroup$

Here is a somewhat adversarial counter-example, where accuracy is better than a proper scoring rule, based on @Benoit_Sanchez's neat thought experiment,

You own an egg shop and each egg you sell generates a net revenue of 2 dollars. Each customer who enters the shop may either buy an egg or leave without buying any. For some customers you can decide to make a discount and you will only get 1 dollar revenue but then the customer will always buy.

You plug a webcam that analyses the customer behaviour with features such as "sniffs the eggs", "holds a book with omelette recipes"... and classify them into "wants to buy at 2 dollars" (positive) and "wants to buy only at 1 dollar" (negative) before he leaves.

If your classifier makes no mistake, then you get the maximum revenue you can expect. If it's not perfect, then:

  • for every false positive you loose 1 dollar because the customer leaves and you didn't try to make a successful discount

  • for every false negative you loose 1 dollar because you make a useless discount

Then the accuracy of your classifier is exactly how close you are to the maximum revenue. It is the perfect measure.

So say we record the amount of time the customer spends "sniffing eggs" and "holding a book with omelette recipes" and make ourselves a classification task:

enter image description here

This is actually my version of Brian Ripley's synthetic benchmark dataset, but lets pretend it is the data for our task. As this is a synthetic task, I can work out the probabilities of class membership according to the true data generating process:

enter image description here

Unfortunately it is upside-down because I couldn't work out how to fix it in MATLAB, but please bear with me. Now in practice, we won't get a perfect model, so here is a model with an error (I have just perturbed the true posterior probabilities with a Gaussian bump)

enter image description here

And here is another one, with a bump in a different place.

enter image description here

Now the Brier score is a proper scoring rule, and it gives a slightly lower (better) score for the second model (because the perturbation is in a region of slightly lower density). However, the perturbation in the first model is well away from the decision boundary, and so that one has a higher accuracy.

Since in this particular application, the accuracy is equal to our financial gain in dollars, the Brier score is selecting the wrong model, and we will lose money.

Vapnik's advice that it is often better to form a purely discriminative classifier directly (rather than estimate a probability and threshold it) is based on this sort of situation. If all we are interested is in making a binary decision, then we don't really care what the classifier does away from the decision boundary, so we shouldn't waste resources modelling features of the data distribution that don't affect the decision.

This is a Laconic "if" though. If it is a classification task with fixed misclassification costs, no covariate shift and known and constant operational class priors, then this approach may indeed be better (and the success of the SVM in many practical applications is some evidence of that). However, many applications are not like that, we may not know ahead of time what the misclassification costs are, or equivalently the operational class frequencies. In those applications we are much better off with a probabilistic classifier, and set the thresholds appropriately according to operational conditions.

Whether accuracy is a good performance metric depends on the needs of the application, there is no "one size fits all" policy. We need to understand the tools we use and be aware of their advantages and pitfalls, and consider the purpose of the exercise in choosing the right tool from the toolbox. In this example, the problem with the Brier score is that it ignores the true needs of the application, and no amount of adjusting the threshold will compensate for it's selection of the wrong model.

It is also important to make a distinction between performance evaluation and model selection - they are not the same thing, and sometimes (often?) it is better to have a proper scoring rule for model selection in order to achieve maximum performance according to your metric of real interest (e.g. accuracy).

$\endgroup$
8
  • $\begingroup$ This answer (+1) usefully shows why accuracy can work when false-negative and false positive costs are equal, for the cost-optimal probability cutoff at 0.5 that accuracy implicitly uses. Much discussion about superiority of proper scoring rules elides the issue of downstream uses and specific cost tradeoffs. If you need to make a choice based on a model, the model needs to work well around the probability cutoff that matches your cost tradeoffs. Different proper scoring rules emphasize different probability regions. Some links are here. $\endgroup$
    – EdM
    Jul 30 at 16:14
  • $\begingroup$ Thanks for the info in the link. Similar examples could be constructed for cases where the misclassification costs were not equal (in which case it would be weighted accuracy - balanced error rate being another special case). I certainly agree with @StephanKolassa that "accuracy is problematic even in the context of balanced datasets", but proper scoring rules are not a panacea. $\endgroup$ Jul 30 at 16:23
  • $\begingroup$ Why do you say this shows that accuracy picks the right model rather than accuracy sometimes picks the right model? There should be some sense of expected value, not just in the occasional instance, should there not? $\endgroup$
    – Dave
    Aug 11 at 18:45
  • $\begingroup$ "Why do you say this shows that accuracy picks the right model rather than accuracy sometimes picks the right model? " I wrote "Here is a somewhat adversarial counter-example, where accuracy is better than a proper scoring rule," which is clearly stating that this is one example where this is the case and is in no way stating a general rule. I even highlighted that it was an adversarial example. $\endgroup$ Aug 11 at 19:07
  • $\begingroup$ But this just shows that sometimes the Brier score picks the wrong model, not any kind of expected loss. Proper scoring rules do not even have to agree on the preferred model. $\endgroup$
    – Dave
    Aug 12 at 11:18
3
$\begingroup$

DaL answer is just exactly this. I'll illustrate it with a very simple example about... selling eggs.

You own an egg shop and each egg you sell generates a net revenue of $2$ dollars. Each customer who enters the shop may either buy an egg or leave without buying any. For some customers you can decide to make a discount and you will only get $1$ dollar revenue but then the customer will always buy.

You plug a webcam that analyses the customer behaviour with features such as "sniffs the eggs", "holds a book with omelette recipes"... and classify them into "wants to buy at $2$ dollars" (positive) and "wants to buy only at $1$ dollar" (negative) before he leaves.

If your classifier makes no mistake, then you get the maximum revenue you can expect. If it's not perfect, then:

  • for every false positive you loose $1$ dollar because the customer leaves and you didn't try to make a successful discount
  • for every false negative you loose $1$ dollar because you make a useless discount

Then the accuracy of your classifier is exactly how close you are to the maximum revenue. It is the perfect measure.

But now if the discount is $a$ dollars. The costs are:

  • false positive: $a$
  • false negative: $2-a$

Then you need an accuracy weighted with these numbers as a measure of efficiency of the classifier. If $a=0.001$ for example, the measure is totally different. This situation is likely related to imbalanced data: few customers are ready to pay $2$, while most would pay $0.001$. You don't care getting many false positives to get a few more true positives. You can adjust the threshold of the classifier according to this.

If the classifier is about finding relevant documents in a database for example, then you can compare "how much" wasting time reading an irrelevant document is compared to finding a relevant document.

$\endgroup$
8
  • $\begingroup$ This is an excellent answer. The performance metric depends on the requirements of the application, and sometimes the accuracy (or more generally the expected loss) is what we are interested in. Of course that doesn't mean it is necessarily the right metric for model selection (e.g. optimising hyper-parameters), but that doesn't mean it shouldn't be used for performance evaluation (or that the "class imbalance problem" is not a problem). $\endgroup$ Jul 27 at 8:32
  • $\begingroup$ This example seems to illustrate my point: it makes most sense to first make probabilistic predictions about the probability a given customer will purchase at a given discount, then make decisions on whether or not to offer a given discount to a given customer. The predictions are one input into the decisions, but there are also others (like the discount amount). Yes, in the end it may turn into a "weighted accuracy", but I'd say separating concerns is clearer. For one, a separation can easily be extended to multi-class problems. $\endgroup$ Jul 27 at 9:06
  • $\begingroup$ @StephanKolassa No, for the problem as stated, accuracy is the quantity of interest. Extending the scenario to include new features, such as variable discount amounts, is avoiding the point made by the analogy. I agree that it is a generally good idea to estimate the probability and then threshold at a suitable level, but the point remains that for this particular scenario, the accuracy is the quantity of interest. $\endgroup$ Jul 30 at 6:56
  • $\begingroup$ The work of Vapnik (especially the SVM) provides justification for solving the classification problem directly (avoids wasting resources, including data, on features that are irrelevant to the decision). For me, whether that is a good idea depends on the requirements of the application. It seems fine for e.g. handwritten digit recognition, where costs and priors are fixed, less so for things like medical diagnosis where operational conditions are more fluid. $\endgroup$ Jul 30 at 6:58
  • $\begingroup$ @DikranMarsupial: accuracy is the KPI of interest here because it happens to lead us to the threshold that maximizes the expected payoff. (I'll trust you two it does, I didn't do the math.) As Benoit notes, if the cost/revenue structure changes, the optimal threshold changes, so we need to optimize a "weighted accuracy". I still believe thinking about this in terms of probabilities and optimal thresholds, separating the statistical and the business problem, is more fruitful. I guess we will not agree here. $\endgroup$ Jul 30 at 6:59
2
$\begingroup$

I wrote a whole blog post on the matter:
https://blog.ephorie.de/zeror-the-simplest-possible-classifier-or-why-high-accuracy-can-be-misleading

ZeroR, the simplest possible classifier, just takes the majority class as the prediction. With highly imbalanced data you will get a very high accuracy, yet if your minority class is the class of interest, this is completely useless. Please find the details and examples in the post.

Bottom line: when dealing with imbalanced data you can construct overly simple classifiers that give a high accuracy yet have no practical value whatsoever...

$\endgroup$
5
  • 1
    $\begingroup$ First, please expand this rather then giving link-only answer. Second, I don't agree with your conclusion, in many cases the data is imbalanced and it may have a lot of practical value, e.g. as discussed in this thread: stats.stackexchange.com/questions/283170/… $\endgroup$
    – Tim
    Apr 28 '20 at 12:01
  • $\begingroup$ @Tim: I don't see any practical value in ignoring the minority class completely (especially not when this is the class of interest!). $\endgroup$
    – vonjd
    Apr 28 '20 at 12:05
  • $\begingroup$ @Tim: Added more details and the gist of the post, hope this helps... Thank you! $\endgroup$
    – vonjd
    Apr 28 '20 at 12:09
  • $\begingroup$ @Tim:...oh, and thank you for the link to the other question, very interesting! $\endgroup$
    – vonjd
    Apr 28 '20 at 12:11
  • 1
    $\begingroup$ Thanks, it's clearer now. $\endgroup$
    – Tim
    Apr 28 '20 at 13:53
1
$\begingroup$

Classification accuracy is the number of correct predictions divided by the total number of predictions.

Accuracy can be misleading. For example, in a problem where there is a large class imbalance, a model can predict the value of the majority class for all predictions and achieve a high classification accuracy. So, further performance measures are needed such as F1 score and Brier score.

$\endgroup$
1
$\begingroup$

After reading through all the answers above, here is an appeal to common sense. Optimality is a flexible term and always needs to be qualified; in other words, saying a model or algorithm is "optimal" is meaningless, especially in a scientific sense.

Whenever anyone says they are scientifically optimizing something, I recommend asking a question like: "In what sense do you define optimality?" This is because in science, unless you can measure something, you cannot optimize (maximize, minimize, etc.) it.

As an example, the OP asks the following:

"Why is accuracy not the best measure for assessing classification models?"

There is an embedded reference to optimization in the word "best" from the question above. "Best" is meaningless in science because "goodness" cannot be measured scientifically.

The scientifically correct response to this question is that the OP needed to define what "good" means. In the real world (outside of academic exercises and Kaggle competitions) there is always a cost/benefit structure to consider when using a machine to suggest or make decisions to or on behalf of/instead of people.

For classification tasks, that information can be embedded in a cost/benefit matrix with entries corresponding to those of the confusion matrix. Finally, since cost/benefit information is a function of the people who are considering using mechanistic help for their decision-making, it is subject to change with the circumstances, and therefore, there is never going to be one fixed measure of optimality which will work for all time in even one problem, let alone all problems (i.e., "models") involving classification.

Any measure of optimality for classification which ignores costs does so at its own risk. Even the ROC AUC fails to be cost-invariant, as shown in this figure.

$\endgroup$
3
  • 3
    $\begingroup$ One of my favorite quotes from a statistician (Youden): "It is, in fact, not a statistical matter to decide what weights should be attached to these two types of diagnostic error." First page of acsjournals.onlinelibrary.wiley.com/doi/abs/10.1002/… $\endgroup$
    – brethvoice
    Mar 28 '20 at 19:00
  • $\begingroup$ Link to my full master's thesis (from which the above linked image was taken): researchgate.net/publication/… $\endgroup$
    – brethvoice
    Oct 14 '20 at 21:18
  • $\begingroup$ After reading the answers above and discussions of "imbalanced classes," I am reminded of N. N. Taleb's writings (e.g., "The Black Swan"). Just sayin... $\endgroup$
    – brethvoice
    Nov 9 '20 at 14:38
-3
$\begingroup$

You may view accuracy as the $R^2$ of classification: an initially appealing metric with which to compare models, that falls short under detailed examination.

In both cases overfitting can be a major problem. Just as in the case of a high $R^2$ might mean that you are modelling the noise rather than the signal, a high accuracy may be a red-flag that your model applied too rigidly to your test dataset and does not have general applicability. This is especially problematic when you have highly imbalanced classification categories. The most accurate model might be a trivial one which classifies all data as one category (with the accuracy equal to proportion of the most frequent category), but this accuracy will fall spectacularly if you need to classify a dataset with a different true distribution of categories.

As others have noted, another problem with accuracy is an implicit indifference to the price of failure - i.e. an assumption that all mis-classifications are equal. In practice they are not, and the costs of getting the wrong classification is highly subject dependent and you may prefer to minimise a particular kind of wrongness than maximise accuracy.

$\endgroup$
2
  • 2
    $\begingroup$ Hum. (1) I'd assume that evaluating accuracy or any other metric out-of-sample would be understood, so I don't really see how accuracy has more of a specific overfitting problem. (2) if you apply a model trained on population A to a different population B, then you are comparing apples to oranges, and I again don't really see how this is a specific problem for accuracy. $\endgroup$ Nov 9 '17 at 11:28
  • 1
    $\begingroup$ (1) It is nevertheless a problem for accuracy, and the question is about using accuracy as a gold-standard. (2) The point of building a classifier is to use it on the oranges, not the just the apples. It should be general enough to capture the essential signals in the data (such that they exist), rather than being a catechism for your training data. $\endgroup$
    – James
    Nov 9 '17 at 11:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.