97
$\begingroup$

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.

Accuracy, the proportion of correct classifications among all classifications, is very simple and very "intuitive" measure, yet it may be a poor measure for imbalanced data. Why does our intuition misguide us here and are there any other problems with this measure?

$\endgroup$
102
+50
$\begingroup$

Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes.

Frank Harrell has written about this on his blog: Classification vs. Prediction and Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules.

Essentially, his argument is that the statistical component of your exercise ends when you output a probability for each class of your new sample. Mapping these predicted probabilities $(\hat{p}, 1-\hat{p})$ to a 0-1 classification, by choosing a threshold beyond which you classify a new observation as 1 vs. 0 is not part of the statistics any more. It is part of the decision component. And here, you need the probabilistic output of your model - but also considerations like:

  • What are the consequences of deciding to treat a new observation as class 1 vs. 0? Do I then send out a cheap marketing mail to all 1s? Or do I apply an invasive cancer treatment with big side effects?
  • What are the consequences of treating a "true" 0 as 1, and vice versa? Will I tick off a customer? Subject someone to unnecessary medical treatment?
  • Are my "classes" truly discrete? Or is there actually a continuum (e.g., blood pressure), where clinical thresholds are in reality just cognitive shortcuts? If so, how far beyond a threshold is the case I'm "classifying" right now?
  • Or does a low-but-positive probability to be class 1 actually mean "get more data", "run another test"?

Depending on the consequences of your decision, you will use a different threshold to make the decision. If the action is invasive surgery, you will require a much higher probability for your classification of the patient as suffering from something than if the action is to recommend two aspirin. Or you might even have three different decisions although there are only two classes (sick vs. healthy): "go home and don't worry" vs. "run another test because the one we have is inconclusive" vs. "operate immediately".

The correct way of assessing predicted probabilities $(\hat{p}, 1-\hat{p})$ is not to compare them to a threshold, map them to $(0,1)$ based on the threshold and then assess the transformed $(0,1)$ classification. Instead, one should use proper . These are loss functions that map predicted probabilities and corresponding observed outcomes to loss values, which are minimized in expectation by the true probabilities $(p,1-p)$. The idea is that we take the average over the scoring rule evaluated on multiple (best: many) observed outcomes and the corresponding predicted class membership probabilities, as an estimate of the expectation of the scoring rule.

Note that "proper" here has a precisely defined meaning - there are improper scoring rules as well as proper scoring rules and finally strictly proper scoring rules. Scoring rules as such are loss functions of predictive densities and outcomes. Proper scoring rules are scoring rules that are minimized in expectation if the predictive density is the true density. Strictly proper scoring rules are scoring rules that are only minimized in expectation if the predictive density is the true density.

As Frank Harrell notes, accuracy is an improper scoring rule. (More precisely, accuracy is not even a scoring rule at all: see my answer to Is accuracy an improper scoring rule in a binary classification setting?) This can be seen, e.g., if we have no predictors at all and just a flip of an unfair coin with probabilities $(0.6,0.4)$. Accuracy is maximized if we classify everything as the first class and completely ignore the 40% probability that any outcome might be in the second class. (Here we see that accuracy is problematic even for balanced classes.) Proper will prefer a $(0.6,0.4)$ prediction to the $(1,0)$ one in expectation. In particular, accuracy is discontinuous in the threshold: moving the threshold a tiny little bit may make one (or multiple) predictions change classes and change the entire accuracy by a discrete amount. This makes little sense.

More information can be found at Frank's two blog posts linked to above, as well as in Chapter 10 of Frank Harrell's Regression Modeling Strategies.

(This is shamelessly cribbed from an earlier answer of mine.)


EDIT. My answer to Example when using accuracy as an outcome measure will lead to a wrong conclusion gives a hopefully illustrative example where maximizing accuracy can lead to wrong decisions even for balanced classes.

$\endgroup$
  • 6
    $\begingroup$ @Tim Frank's point (that he discussed in numerous answers on our site and elsewhere), as I understand it, is that if a classification algorithm does not return probabilities then it's garbage and should not be used. To be honest, most of the commonly used algorithms do return probabilities. $\endgroup$ – amoeba Nov 9 '17 at 9:17
  • 5
    $\begingroup$ I'd say that an algorithm that takes past observations and outputs only classifications without taking the points above into account (e.g., costs of mis-decisions) conflates the statistical and the decision aspect. It's like someone recommending a particular type of car to you without first asking you whether you want to transport a little league baseball team, a bunch of building materials, or only yourself. So I'd also say such an algorithm would be garbage. $\endgroup$ – Stephan Kolassa Nov 9 '17 at 9:23
  • 7
    $\begingroup$ I was going to write an answer, but then didn't need to. Bravo. I discuss this with my students as a "separation of concerns" between statistical modeling and decision making. This type of concept is very deeply rooted in engineering culture. $\endgroup$ – Matthew Drury Nov 9 '17 at 14:34
  • 7
    $\begingroup$ @chainD: if your classifier (remember, it's the one with the highest accuracy) says that "everyone in this sample is healthy", then what doctor or analyst would believe that there is more to the story? I agree that in the end, it's a call for the analyst to make, but "everyone is healthy" is far less helpful to the analyst than something that draws attention to residual uncertainty like the 95%/5% prediction. $\endgroup$ – Stephan Kolassa Nov 10 '17 at 20:43
  • 11
    $\begingroup$ @StephanKolassa 's answer and comments are superb. Someone else comment implied that there is a difference in how this is viewed depending on which culture you are part of. This is not really the case; it's just that some fields bothered to understand the literature and others didn't. Weather forecasting, for example, has been at the forefront and has used proper scoring rules for assessing forecaster accuracy since at least 1951. $\endgroup$ – Frank Harrell Nov 11 '17 at 12:53
76
$\begingroup$

When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a great way to lower the cost. Predict that every instance belongs to the majority class, get accuracy of 99% and go home early.

The problem starts when the actual costs that we assign to every error are not equal. If we deal with a rare but fatal disease, the cost of failing to diagnose the disease of a sick person is much higher than the cost of sending a healthy person to more tests.

In general, there is no general best measure. The best measure is derived from your needs. In a sense, it is not a machine learning question, but a business question. It is common that two people will use the same data set but will choose different metrics due to different goals.

Accuracy is a great metric. Actually, most metrics are great and I like to evaluate many metrics. However, at some point you will need to decide between using model A or B. There you should use a single metric that best fits your need.

For extra credit, choose this metric before the analysis, so you won't be distracted when making the decision.

$\endgroup$
  • 3
    $\begingroup$ Great answer - I've proposed a couple of edits just to try and make the point clearer to beginners in machine learning (at whom this question is aimed). $\endgroup$ – nekomatic Nov 9 '17 at 8:47
  • 1
    $\begingroup$ I'd disagree that it's not a machine learning problem. But addressing it would involve doing machine learning on the meta problem and necessitate the machine having access to some kind of data beyond just the basic classification information. $\endgroup$ – Shufflepants Nov 9 '17 at 20:41
  • 3
    $\begingroup$ I don't see it as a function of only the data since different goals can lad to different cost/model/performance/metrics. I do agree that in general, the question of cost can be handled mathematically. However questions like the cost of treating patients rely on totally different information. This information needed for the meta data is usually not suitable for machine learning methodology so most of the time it is handled with different methods. $\endgroup$ – DaL Nov 12 '17 at 12:01
  • 2
    $\begingroup$ By "misdiagnosing a person with the disease", you mean "misdiagnosing a person who has the disease (as not having the disease)", right? Because that phrase could be interpreted either way. $\endgroup$ – Tanner Swett Nov 13 '17 at 1:39
  • $\begingroup$ You are right Tanner. I changed the test to make it clearer. $\endgroup$ – DaL Nov 13 '17 at 6:52
20
$\begingroup$

The problem with accuracy

Standard accuracy is defined as the ratio of correct classifications to the number of classifications done.

\begin{align*} accuracy := \frac{\text{correct classifications}}{\text{number of classifications}} \end{align*}

It is thus on overall measure over all classes and as we'll shortly see it's not a good measure to tell an oracle apart from an actual useful test. An oracle is a classification function that returns a random guess for each sample. Likewise, we want to be able to rate the classification performance of our classification function. Accuracy \textit{can} be a useful measure if we have the same amount of samples per class but if we have an imbalanced set of samples accuracy isn't useful at all. Even more so, a test can have a high accuracy but actually perform worse than a test with a lower accuracy.

If we have a distribution of samples such that 90\% of samples belong to class $\mathcal{A}$, 5\% belonging to $\mathcal{B}$ and another 5\% belonging to $\mathcal{C}$ then the following classification function will have an accuracy of $0.9$:

\begin{align*} classify(sample) := \begin{cases} \mathcal{A} & \text{if }\top \\ \end{cases} \end{align*}

Yet, it is obvious given that we know how $classify$ works that this it can not tell the classes apart at all. Likewise, we can construct a classification function

\begin{align*} classify(sample) := \text{guess} \begin{cases} \mathcal{A} & \text{with p } = 0.96 \\ \mathcal{B} & \text{with p } = 0.02 \\ \mathcal{C} & \text{with p } = 0.02 \\ \end{cases} \end{align*}

which has an accuracy of $0.96 \cdot 0.9 + 0.02 \cdot 0.05 \cdot 2 = 0.866$ and will not always predict $\mathcal{A}$ but still given that we know how $classify$ works it is obvious that it can not tell classes apart. Accuracy in this case only tells us how good our classification function is at guessing. This means that accuracy is not a good measure to tell an oracle apart from a useful test.

Accuracy per Class

We can compute the accuracy individually per class by giving our classification function only samples from the same class and remember and count the number of correct classifications and incorrect classifications then compute $accuracy := \text{correct}/(\text{correct} + \text{incorrect})$. We repeat this for every class. If we have a classification function that can accurately recognize class $\mathcal{A}$ but will output a random guess for the other classes then this results in an accuracy of $1.00$ for $\mathcal{A}$ and an accuracy of $0.33$ for the other classes. This already provides us a much better way to judge the performance of our classification function. An oracle always guessing the same class will produce a per class accuracy of $1.00$ for that class, but $0.00$ for the other class. If our test is useful all the accuracies per class should be $>0.5$. Otherwise, our test isn't better than chance. However, accuracy per class does not take into account false positives. Even though our classification function has a 100\% accuracy for class $\mathcal{A}$ there will also be false positives for $\mathcal{A}$ (such as a $\mathcal{B}$ wrongly classified as a $\mathcal{A}$).

Sensitivity and Specificity

In medical tests sensitivity is defined as the ratio between people correctly identified as having the disease and the amount of people actually having the disease. Specificity is defined as the ratio between people correctly identified as healthy and the amount of people that are actually healthy. The amount of people actually having the disease is the amount of true positive test results plus the amount of false negative test results. The amount of actually healthy people is the amount of true negative test results plus the amount of false positive test results.

Binary Classification

In binary classification problems there are two classes $\mathcal{P}$ and $\mathcal{N}$. $T_{n}$ refers to the number of samples that were correctly identified as belonging to class $n$ and $F_{n}$ refers to the number of samples that werey falsely identified as belonging to class $n$. In this case sensitivity and specificity are defined as following:

\begin{align*} sensitivity := \frac{T_{\mathcal{P}}}{T_{\mathcal{P}}+F_{\mathcal{N}}} \\ specificity := \frac{T_{\mathcal{N}}}{T_{\mathcal{N}}+F_{\mathcal{P}}} \end{align*}

$T_{\mathcal{P}}$ being the true positives $F_{\mathcal{N}}$ being the false negatives, $T_{\mathcal{N}}$ being the true negatives and $F_{\mathcal{P}}$ being the false positives. However, thinking in terms of negatives and positives is fine for medical tests but in order to get a better intuition we should not think in terms of negatives and positives but in generic classes $\alpha$ and $\beta$. Then, we can say that the amount of samples correctly identified as belonging to $\alpha$ is $T_{\alpha}$ and the amount of samples that actually belong to $\alpha$ is $T_{\alpha} + F_{\beta}$. The amount of samples correctly identified as not belonging to $\alpha$ is $T_{\beta}$ and the amount of samples actually not belonging to $\alpha$ is $T_{\beta} + F_{\alpha}$. This gives us the sensitivity and specificity for $\alpha$ but we can also apply the same thing to the class $\beta$. The amount of samples correctly identified as belonging to $\beta$ is $T_{\beta}$ and the amount of samples actually belonging to $\beta$ is $T_{\beta} + F_{\alpha}$. The amount of samples correctly identified as not belonging to $\beta$ is $T_{\alpha}$ and the amount of samples actually not belonging to $\beta$ is $T_{\alpha} + F_{\beta}$. We thus get a sensitivity and specificity per class:

\begin{align*} sensitivity_{\alpha} := \frac{T_{\alpha}}{T_{\alpha}+F_{\beta}} \\ specificity_{\alpha} := \frac{T_{\beta}}{T_{\beta} + F_{\alpha}} \\ sensitivity_{\beta} := \frac{T_{\beta}}{T_{\beta}+F_{\alpha}} \\ specificity_{\beta} := \frac{T_{\alpha}}{T_{\alpha} + F_{\beta}} \\ \end{align*}

We however observe that $sensitivity_{\alpha} = specificity_{\beta}$ and $specificity_{\alpha} = sensitivity_{\beta}$. This means that if we only have two classes we don't need sensitivity and specificity per class.

N-Ary Classification

Sensitivity and specificity per class isn't useful if we only have two classes, but we can extend it to multiple classes. Sensitivity and specificity is defined as:

\begin{align*} \text{sensitivity} := \frac{\text{true positives}}{\text{true positives} + \text{false negatives}} \\ \text{specificity} := \frac{\text{true negatives}}{\text{true negatives} + \text{false-positives}} \\ \end{align*}

The true positives is simply $T_{n}$, the false negatives is simply $\sum_{i}(F_{n,i})$ and the false positives is simply $\sum_{i}(F_{i,n})$. Finding the true negatives is much harder but we can say that if we correctly classify something as belonging to a class different than $n$ it counts as a true negative. This means we have at least $\sum_{i}(T_{i}) - T(n)$ true negatives. However, this aren't all true negatives. All the wrong classifications for a class different than $n$ are also true negatives, because they correctly weren't identified as belonging to $n$. $\sum_{i}(\sum_{k}(F_{i,k}))$ represents all wrong classifications. From this we have to subtract the cases where the input class was $n$ meaning we have to subtract the false negatives for $n$ which is $\sum_{i}(F_{n,i})$ but we also have to subtract the false positives for $n$ because they are false positives and not true negatives so we have to also subtract $\sum_{i}(F_{i,n})$ finally getting $\sum_{i}(T_{i}) - T(n) + \sum_{i}(\sum_{k}(F_{n,i})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n})$. As a summary we have:

\begin{align*} \text{true positives} := T_{n} \\ \text{true negatives} := \sum_{i}(T_{i}) - T(n) + \sum_{i}(\sum_{k}(F_{n,i})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n}) \\ \text{false positives} := \sum_{i}(F_{i,n}) \\ \text{false negatives} := \sum_{i}(F_{n,i}) \end{align*}

\begin{align*} sensitivity(n) := \frac{T_{n}}{T_{n} + \sum_{i}(F_{n,i})} \\ specificity(n) := \frac{\sum_{i}(T_{i}) - T_{n} + \sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{n,i}) - \sum_{i}(F_{i,n})}{\sum_{i}(T_{i}) - T_{n} + \sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{n,i})} \end{align*}

Introducing Confidence

We define a $confidence^{\top}$ which is a measure of how confident we can be that the reply of our classification function is actually correct. $T_{n} + \sum_{i}(F_{i,n})$ are all cases where the classification function replied with $n$ but only $T_{n}$ of those are correct. We thus define

\begin{align*} confidence^{\top}(n) := \frac{T_{n}}{T_{n}+\sum_{i}(F_{i,n})} \end{align*}

But can we also define a $confidence^{\bot}$ which is a measure of how confident we can be that if our classification function responds with a class different than $n$ that it actually wasn't an $n$?

Well, we get $\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}$ all of which are correct except $\sum_{i}(F_{n,i})$.Thus, we define

\begin{align*} confidence^{\bot}(n) = \frac{\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}-\sum_{i}(F_{n,i})}{\sum_{i}(\sum_{k}(F_{i,k})) - \sum_{i}(F_{i,n}) + \sum_{i}(T_{i}) - T_{n}} \end{align*}

$\endgroup$
  • $\begingroup$ Can you please provide any example of calculating Mean Accuracy using confusion matrix. $\endgroup$ – Aadnan Farooq A Sep 27 '18 at 1:17
  • $\begingroup$ You can find a more detailed description with examples here: mroman.ch/guides/sensspec.html $\endgroup$ – mroman Sep 27 '18 at 14:40
  • $\begingroup$ Reading through it again there's an error in the definition of confidence_false. I'm surprised nobody spotted that. I'll fix that in the next few days. $\endgroup$ – mroman Sep 27 '18 at 16:14
8
$\begingroup$

Imbalanced classes in your dataset

To be short: imagine, 99% of one class (say apples) and 1% of another class is in your data set (say bananas). My super duper algorithm gets an astonishing 99% accuracy for this data set, check it out:

return "it's an apple"

He will be right 99% of the time and therefore gets a 99% accuracy. Can I sell you my algorithm?

Solution: don't use an absolute measure (accuracy) but a relative-to-each-class measure (there are a lot out there, like ROC AUC)

$\endgroup$
2
$\begingroup$

DaL answer is just exactly this. I'll illustrate it with a very simple example about... selling eggs.

You own an egg shop and each egg you sell generates a net revenue of $2$ dollars. Each customer who enters the shop may either buy an egg or leave without buying any. For some customers you can decide to make a discount and you will only get $1$ dollar revenue but then the customer will always buy.

You plug a webcam that analyses the customer behaviour with features such as "sniffs the eggs", "holds a book with omelette recipes"... and classify them into "wants to buy at $2$ dollars" (positive) and "wants to buy only at $1$ dollar" (negative) before he leaves.

If your classifier makes no mistake, then you get the maximum revenue you can expect. If it's not perfect, then:

  • for every false positive you loose $1$ dollar because the customer leaves and you didn't try to make a successful discount
  • for every false negative you loose $1$ dollar because you make a useless discount

Then the accuracy of your classifier is exactly how close you are to the maximum revenue. It is the perfect measure.

But now if the discount is $a$ dollars. The costs are:

  • false positive: $a$
  • false negative: $2-a$

Then you need an accuracy weighted with these numbers as a measure of efficiency of the classifier. If $a=0.001$ for example, the measure is totally different. This situation is likely related to imbalanced data: few customers are ready to pay $2$, while most would pay $0.001$. You don't care getting many false positives to get a few more true positives. You can adjust the threshold of the classifier according to this.

If the classifier is about finding relevant documents in a database for example, then you can compare "how much" wasting time reading an irrelevant document is compared to finding a relevant document.

$\endgroup$
1
$\begingroup$

Classification accuracy is the number of correct predictions divided by the total number of predictions.

Accuracy can be misleading. For example, in a problem where there is a large class imbalance, a model can predict the value of the majority class for all predictions and achieve a high classification accuracy. So, further performance measures are needed such as F1 score and Brier score.

$\endgroup$
-3
$\begingroup$

You may view accuracy as the $R^2$ of classification: an initially appealing metric with which to compare models, that falls short under detailed examination.

In both cases overfitting can be a major problem. Just as in the case of a high $R^2$ might mean that you are modelling the noise rather than the signal, a high accuracy may be a red-flag that your model applied too rigidly to your test dataset and does not have general applicability. This is especially problematic when you have highly imbalanced classification categories. The most accurate model might be a trivial one which classifies all data as one category (with the accuracy equal to proportion of the most frequent category), but this accuracy will fall spectacularly if you need to classify a dataset with a different true distribution of categories.

As others have noted, another problem with accuracy is an implicit indifference to the price of failure - i.e. an assumption that all mis-classifications are equal. In practice they are not, and the costs of getting the wrong classification is highly subject dependent and you may prefer to minimise a particular kind of wrongness than maximise accuracy.

$\endgroup$
  • 2
    $\begingroup$ Hum. (1) I'd assume that evaluating accuracy or any other metric out-of-sample would be understood, so I don't really see how accuracy has more of a specific overfitting problem. (2) if you apply a model trained on population A to a different population B, then you are comparing apples to oranges, and I again don't really see how this is a specific problem for accuracy. $\endgroup$ – Stephan Kolassa Nov 9 '17 at 11:28
  • $\begingroup$ (1) It is nevertheless a problem for accuracy, and the question is about using accuracy as a gold-standard. (2) The point of building a classifier is to use it on the oranges, not the just the apples. It should be general enough to capture the essential signals in the data (such that they exist), rather than being a catechism for your training data. $\endgroup$ – James Nov 9 '17 at 11:45

protected by kjetil b halvorsen Nov 12 '17 at 20:09

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.