Skip to main content
241 votes

Why is accuracy not the best measure for assessing classification models?

Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes. Frank Harrell has written about ...
Stephan Kolassa's user avatar
137 votes

What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

Shortcomings of the MAPE The MAPE, as a percentage, only makes sense for values where divisions and ratios make sense. It doesn't make sense to calculate percentages of temperatures, for instance, so ...
Stephan Kolassa's user avatar
121 votes
Accepted

F1/Dice-Score vs IoU

You're on the right track. So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other: $$ F/2 \leq IoU \leq ...
willem's user avatar
  • 1,326
111 votes

Why is accuracy not the best measure for assessing classification models?

When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a ...
DaL's user avatar
  • 4,692
84 votes
Accepted

Why do I get a 100% accuracy decision tree?

Your test sample is a subset of your training sample: x_train = x[0:2635] x_test = x[0:658] y_train = y[0:2635] y_test = y[0:658] This means that you evaluate ...
Stephan Kolassa's user avatar
73 votes

What is the best way to remember the difference between sensitivity, specificity, precision, accuracy, and recall?

For precision and recall, each is the true positive (TP) as the numerator divided by a different denominator. Precision: TP / Predicted positive Recall: TP / Real positive
beardc's user avatar
  • 849
45 votes

Good accuracy despite high loss value

I have experienced a similar issue. I have trained my neural network binary classifier with a cross entropy loss. Here the result of the cross entropy as a function of epoch. Red is for the training ...
ahstat's user avatar
  • 1,250
36 votes
Accepted

Is an overfitted model necessarily useless?

I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or ...
Richard Hardy's user avatar
35 votes
Accepted

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
Peter Flom's user avatar
  • 125k
34 votes

Is the Dice coefficient the same as accuracy?

The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
Dvir Berebi's user avatar
33 votes

Is an overfitted model necessarily useless?

In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general ...
Haitao Du's user avatar
  • 37.2k
33 votes
Accepted

Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?

This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on. This question can almost certainly not be answered well for you by readers at ...
mkt's user avatar
  • 18.9k
32 votes

Why is accuracy not the best measure for assessing classification models?

The problem with accuracy Standard accuracy is defined as the ratio of correct classifications to the number of classifications done. \begin{align*} accuracy := \frac{\text{correct classifications}}{...
mroman's user avatar
  • 471
32 votes

Is accuracy an improper scoring rule in a binary classification setting?

TL;DR Accuracy is an improper scoring rule. Don't use it. The slightly longer version Actually, accuracy is not even a scoring rule. So asking whether it is (strictly) proper is a category error. ...
Stephan Kolassa's user avatar
30 votes
Accepted

How to determine the accuracy of regression? Which measure should be used?

You should ask yourself what were you trying to achieve with your modeling approach. As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for ...
Firebug's user avatar
  • 19.5k
27 votes
Accepted

Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

I guess I'm one of the "among others", so I'll chime in. The short version: I'm afraid your example is a bit of a straw man, and I don't think we can learn a lot from it. In the first case, yes, you ...
Stephan Kolassa's user avatar
26 votes

Example when using accuracy as an outcome measure will lead to a wrong conclusion

I'll cheat. Specifically, I have argued often (e.g., here) that the statistical part of modeling and prediction extends only to making probabilistic predictions for class memberships (or giving ...
Stephan Kolassa's user avatar
22 votes

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical ...
call-in-co's user avatar
  • 1,016
21 votes

Why do I get a 100% accuracy decision tree?

You are getting 100% accuracy because you are using a part of training data for testing. At the time of training, decision tree gained the knowledge about that data, and now if you give same data to ...
Sanjay Chandlekar's user avatar
21 votes

Why not use evaluation metrics as the loss function?

Maximizing accuracy (percent of correctly examples) is the same as minimizing error rate (percent of incorrectly classified examples). For a single observation, the loss function for the error rate is ...
Sycorax's user avatar
  • 92.5k
21 votes
Accepted

Getting 99-100% accuracy on my training/validation data but performs bad on completely new data

It shows pretty well that in general, it is important to read datasets documentation and understand their overall context. Here, it seems you're talking about the "ASL Alphabet" dataset ...
J-J-J's user avatar
  • 4,915
20 votes

Why is accuracy not the best measure for assessing classification models?

Here is a somewhat adversarial counter-example, where accuracy is better than a proper scoring rule, based on @Benoit_Sanchez's neat thought experiment, You own an egg shop and each egg you sell ...
Dikran Marsupial's user avatar
20 votes

Why do I get a 100% accuracy decision tree?

As other users have told you, you are using as test set a subset of the train set, and a decision tree is very prone to overfitting. You almost had it when you imported ...
Juan Ignacio Gil's user avatar
19 votes

Good accuracy despite high loss value

One important thing to note as well is that the cross entropy is not a bounded loss. Which means that a single very wrong prediction can potentially make your loss "blow up". In that sense it is ...
Manu's user avatar
  • 291
18 votes
Accepted

How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy?

We can't tell you. It depends on your situation and how easy prediction is in your situation. How many coin tosses do you need to observe before you can predict the next one with 90% accuracy? Related:...
Stephan Kolassa's user avatar
18 votes

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

The main one that springs to mind is "Three myths about risk thresholds for prediction models" by Wynants et al. (2019) where they argue strongly against using a "universally optimal ...
usεr11852's user avatar
  • 44.9k
16 votes

How can we judge the accuracy of Nate Silver's predictions?

Probabilistic forecasts (or, as they are also known, density forecasts) can be evaluated using scoring-rules, i.e., functions that map a density forecast and an observed outcome to a so-called score, ...
Stephan Kolassa's user avatar
16 votes
Accepted

Why does the accuracy not change, when applying different alpha values in L2 regularisation

The various loss functions reflect different answers to the question What makes a model "good"? Choosing one loss function over another is implicitly choosing one interpretation of "...
Sycorax's user avatar
  • 92.5k
16 votes

Is an overfitted model necessarily useless?

Maybe: beware. When you say that 70% accuracy (however you measure it) is good enough for you, it feels like you're assuming that errors are randomly or evenly distributed. But one of the ways of ...
Wayne's user avatar
  • 21.5k
16 votes

Why is accuracy not the best measure for assessing classification models?

Imbalanced classes in your dataset To be short: imagine, 99% of one class (say apples) and 1% of another class is in your data set (say bananas). My super duper algorithm gets an astonishing 99% ...
Mayou36's user avatar
  • 1,165

Only top scored, non community-wiki answers of a minimum length are eligible