# Tag Info

### Why is accuracy not the best measure for assessing classification models?

Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes. Frank Harrell has written about ...
• 127k

### What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

Shortcomings of the MAPE The MAPE, as a percentage, only makes sense for values where divisions and ratios make sense. It doesn't make sense to calculate percentages of temperatures, for instance, so ...
• 127k
Accepted

### F1/Dice-Score vs IoU

You're on the right track. So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other:  F/2 \leq IoU \leq ...
• 1,326

### Why is accuracy not the best measure for assessing classification models?

When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a ...
• 4,692
Accepted

### Why do I get a 100% accuracy decision tree?

Your test sample is a subset of your training sample: x_train = x[0:2635] x_test = x[0:658] y_train = y[0:2635] y_test = y[0:658] This means that you evaluate ...
• 127k

### What is the best way to remember the difference between sensitivity, specificity, precision, accuracy, and recall?

For precision and recall, each is the true positive (TP) as the numerator divided by a different denominator. Precision: TP / Predicted positive Recall: TP / Real positive
• 849

### Good accuracy despite high loss value

I have experienced a similar issue. I have trained my neural network binary classifier with a cross entropy loss. Here the result of the cross entropy as a function of epoch. Red is for the training ...
• 1,250
Accepted

### Is an overfitted model necessarily useless?

I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or ...
• 68.6k
Accepted

### Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
• 125k

### Is the Dice coefficient the same as accuracy?

The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true ...
• 461

### Is an overfitted model necessarily useless?

In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general ...
• 37.2k
Accepted

### Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?

This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on. This question can almost certainly not be answered well for you by readers at ...
• 18.9k

### Why is accuracy not the best measure for assessing classification models?

The problem with accuracy Standard accuracy is defined as the ratio of correct classifications to the number of classifications done. \begin{align*} accuracy := \frac{\text{correct classifications}}{...
• 471

### Is accuracy an improper scoring rule in a binary classification setting?

TL;DR Accuracy is an improper scoring rule. Don't use it. The slightly longer version Actually, accuracy is not even a scoring rule. So asking whether it is (strictly) proper is a category error. ...
• 127k
Accepted

### How to determine the accuracy of regression? Which measure should be used?

You should ask yourself what were you trying to achieve with your modeling approach. As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for ...
• 19.5k
Accepted

### Proper scoring rule when there is a decision to make (e.g. spam vs ham email)

I guess I'm one of the "among others", so I'll chime in. The short version: I'm afraid your example is a bit of a straw man, and I don't think we can learn a lot from it. In the first case, yes, you ...
• 127k

### Example when using accuracy as an outcome measure will lead to a wrong conclusion

I'll cheat. Specifically, I have argued often (e.g., here) that the statistical part of modeling and prediction extends only to making probabilistic predictions for class memberships (or giving ...
• 127k

### Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical ...
• 1,016

### Why do I get a 100% accuracy decision tree?

You are getting 100% accuracy because you are using a part of training data for testing. At the time of training, decision tree gained the knowledge about that data, and now if you give same data to ...

### Why not use evaluation metrics as the loss function?

Maximizing accuracy (percent of correctly examples) is the same as minimizing error rate (percent of incorrectly classified examples). For a single observation, the loss function for the error rate is ...
• 92.5k
Accepted

### Getting 99-100% accuracy on my training/validation data but performs bad on completely new data

It shows pretty well that in general, it is important to read datasets documentation and understand their overall context. Here, it seems you're talking about the "ASL Alphabet" dataset ...
• 4,915

### Why is accuracy not the best measure for assessing classification models?

Here is a somewhat adversarial counter-example, where accuracy is better than a proper scoring rule, based on @Benoit_Sanchez's neat thought experiment, You own an egg shop and each egg you sell ...
• 55.1k

### Why do I get a 100% accuracy decision tree?

As other users have told you, you are using as test set a subset of the train set, and a decision tree is very prone to overfitting. You almost had it when you imported ...

### Good accuracy despite high loss value

One important thing to note as well is that the cross entropy is not a bounded loss. Which means that a single very wrong prediction can potentially make your loss "blow up". In that sense it is ...
• 291
Accepted

### How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy?

We can't tell you. It depends on your situation and how easy prediction is in your situation. How many coin tosses do you need to observe before you can predict the next one with 90% accuracy? Related:...
• 127k

### Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

The main one that springs to mind is "Three myths about risk thresholds for prediction models" by Wynants et al. (2019) where they argue strongly against using a "universally optimal ...
• 44.9k

### How can we judge the accuracy of Nate Silver's predictions?

Probabilistic forecasts (or, as they are also known, density forecasts) can be evaluated using scoring-rules, i.e., functions that map a density forecast and an observed outcome to a so-called score, ...
• 127k
Accepted

### Why does the accuracy not change, when applying different alpha values in L2 regularisation

The various loss functions reflect different answers to the question What makes a model "good"? Choosing one loss function over another is implicitly choosing one interpretation of "...
• 92.5k