244
votes
Why is accuracy not the best measure for assessing classification models?
Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes.
Frank Harrell has written about ...
- 131k
140
votes
What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?
Shortcomings of the MAPE
The MAPE, as a percentage, only makes sense for values where divisions and ratios make sense. It doesn't make sense to calculate percentages of temperatures, for instance, so ...
- 131k
123
votes
Accepted
F1/Dice-Score vs IoU
You're on the right track.
So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other:
$$ F/2 \leq IoU \leq ...
- 1,346
111
votes
Why is accuracy not the best measure for assessing classification models?
When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a ...
- 4,732
84
votes
Accepted
Why do I get a 100% accuracy decision tree?
Your test sample is a subset of your training sample:
x_train = x[0:2635]
x_test = x[0:658]
y_train = y[0:2635]
y_test = y[0:658]
This means that you evaluate ...
- 131k
45
votes
Good accuracy despite high loss value
I have experienced a similar issue.
I have trained my neural network binary classifier with a cross entropy loss. Here the result of the cross entropy as a function of epoch. Red is for the training ...
- 1,260
36
votes
Accepted
Is an overfitted model necessarily useless?
I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or ...
- 69.5k
35
votes
Accepted
Are there parameters where a biased estimator is considered "better" than the unbiased estimator?
One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
- 128k
34
votes
Is the Dice coefficient the same as accuracy?
The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
- 461
34
votes
Accepted
Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?
This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on.
This question can almost certainly not be answered well for you by readers at ...
- 20.4k
33
votes
Is an overfitted model necessarily useless?
In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general ...
- 37.3k
33
votes
Why is accuracy not the best measure for assessing classification models?
The problem with accuracy
Standard accuracy is defined as the ratio of correct classifications to the number of classifications done.
\begin{align*}
accuracy := \frac{\text{correct classifications}}{...
- 481
33
votes
Is accuracy an improper scoring rule in a binary classification setting?
TL;DR
Accuracy is an improper scoring rule. Don't use it.
The slightly longer version
Actually, accuracy is not even a scoring rule. So asking whether it is (strictly) proper is a category error. ...
- 131k
30
votes
Accepted
How to determine the accuracy of regression? Which measure should be used?
You should ask yourself what were you trying to achieve with your modeling approach.
As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for ...
- 19.5k
28
votes
Accepted
Proper scoring rule when there is a decision to make (e.g. spam vs ham email)
I guess I'm one of the "among others", so I'll chime in.
The short version: I'm afraid your example is a bit of a straw man, and I don't think we can learn a lot from it.
In the first case, yes, you ...
- 131k
27
votes
Example when using accuracy as an outcome measure will lead to a wrong conclusion
I'll cheat.
Specifically, I have argued often (e.g., here) that the statistical part of modeling and prediction extends only to making probabilistic predictions for class memberships (or giving ...
- 131k
22
votes
Are there parameters where a biased estimator is considered "better" than the unbiased estimator?
Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization).
As for a mathematical ...
- 1,046
22
votes
Accepted
Getting 99-100% accuracy on my training/validation data but performs bad on completely new data
It shows pretty well that in general, it is important to read datasets documentation and understand their overall context. Here, it seems you're talking about the "ASL Alphabet" dataset ...
- 5,923
21
votes
Why is accuracy not the best measure for assessing classification models?
Here is a somewhat adversarial counter-example, where accuracy is better than a proper scoring rule, based on @Benoit_Sanchez's neat thought experiment,
You own an egg shop and each egg you sell ...
- 56.2k
21
votes
Why do I get a 100% accuracy decision tree?
You are getting 100% accuracy because you are using a part of training data for testing. At the time of training, decision tree gained the knowledge about that data, and now if you give same data to ...
21
votes
Why not use evaluation metrics as the loss function?
Maximizing accuracy (percent of correctly examples) is the same as minimizing error rate (percent of incorrectly classified examples). For a single observation, the loss function for the error rate is ...
- 94.1k
20
votes
Why do I get a 100% accuracy decision tree?
As other users have told you, you are using as test set a subset of the train set, and a decision tree is very prone to overfitting.
You almost had it when you imported
...
- 301
19
votes
Good accuracy despite high loss value
One important thing to note as well is that the cross entropy is not a bounded loss. Which means that a single very wrong prediction can potentially make your loss "blow up". In that sense it is ...
- 291
18
votes
Accepted
How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy?
We can't tell you. It depends on your situation and how easy prediction is in your situation.
How many coin tosses do you need to observe before you can predict the next one with 90% accuracy?
Related:...
- 131k
18
votes
Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity
The main one that springs to mind is "Three myths about risk thresholds for prediction models" by Wynants et al. (2019) where they argue strongly against using a "universally optimal ...
- 46.1k
16
votes
How can we judge the accuracy of Nate Silver's predictions?
Probabilistic forecasts (or, as they are also known, density forecasts) can be evaluated using scoring-rules, i.e., functions that map a density forecast and an observed outcome to a so-called score, ...
- 131k
16
votes
Accepted
Why does the accuracy not change, when applying different alpha values in L2 regularisation
The various loss functions reflect different answers to the question What makes a model "good"? Choosing one loss function over another is implicitly choosing one interpretation of "...
- 94.1k
16
votes
Is an overfitted model necessarily useless?
Maybe: beware. When you say that 70% accuracy (however you measure it) is good enough for you, it feels like you're assuming that errors are randomly or evenly distributed.
But one of the ways of ...
- 21.6k
16
votes
Why is accuracy not the best measure for assessing classification models?
Imbalanced classes in your dataset
To be short: imagine, 99% of one class (say apples) and 1% of another class is in your data set (say bananas). My super duper algorithm gets an astonishing 99% ...
- 1,164
15
votes
sklearn.metrics.accuracy_score vs. LogisticRegression().score?
I wish I could just take this back...amazing what happens when you put your confusion down in writing (and read the source code).
One is testing accuracy, the other is training accuracy.
To clarify: ...
- 361
Only top scored, non community-wiki answers of a minimum length are eligible