Tag Info

179

Most of the other answers focus on the example of unbalanced classes. Yes, this is important. However, I argue that accuracy is problematic even with balanced classes. Frank Harrell has written about this on his blog: Classification vs. Prediction and Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules. ...

100

When we use accuracy, we assign equal cost to false positives and false negatives. When that data set is imbalanced - say it has 99% of instances in one class and only 1 % in the other - there is a great way to lower the cost. Predict that every instance belongs to the majority class, get accuracy of 99% and go home early. The problem starts when the actual ...

95

Shortcomings of the MAPE The MAPE, as a percentage, only makes sense for values where divisions and ratios make sense. It doesn't make sense to calculate percentages of temperatures, for instance, so you shouldn't use the MAPE to calculate the accuracy of a temperature forecast. If just a single actual is zero, $A_t=0$, then you divide by zero in ...

89

You're on the right track. So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other: $$F/2 \leq IoU \leq F$$ and also that they meet at the extremes of one and zero under the conditions that you would expect (perfect match and completely disjoint). Note also that ...

81

Your test sample is a subset of your training sample: x_train = x[0:2635] x_test = x[0:658] y_train = y[0:2635] y_test = y[0:658] This means that you evaluate your model on a part of your training data, i.e., you are doing in-sample evaluation. In-sample accuracy is a notoriously poor indicator to out-of-sample accuracy, and maximizing in-sample accuracy ...

56

For precision and recall, each is the true positive (TP) as the numerator divided by a different denominator. Precision: TP / Predicted positive Recall: TP / Real positive

35

I have experienced a similar issue. I have trained my neural network binary classifier with a cross entropy loss. Here the result of the cross entropy as a function of epoch. Red is for the training set and blue is for the test set. By showing the accuracy, I had the surprise to get a better accuracy for epoch 1000 compared to epoch 50, even for the test ...

35

I think the argument is correct. If 70% is acceptable in the particular application, then the model is useful even though it is overfitted (more generally, regardless of whether it is overfitted or not). While balancing overfitting against underfitting concerns optimality (looking for an optimal solution), having satisfactory performance is about ...

34

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g. install.packages("ridge") library(ridge) set.seed(831) data(GenCont) ridgemod <- linearRidge(Phenotypes ~ ., data = ...

33

In my past project with Credit Card Fraud detection, we intentionally want to over fit the data / hard coded to remember fraud cases. (Note, overfitting one class is not exactly the general overfitting problem OP talked about.) Such system has relatively low false positives and satisfy our needs. So, I would say, overfitted model can be useful for some ...

28

The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true negatives while Dice coefficient and many other measures just handle true negatives as uninteresting defaults (see The Basics of Classifier Evaluation, Part 1). ...

28

The problem with accuracy Standard accuracy is defined as the ratio of correct classifications to the number of classifications done. \begin{align*} accuracy := \frac{\text{correct classifications}}{\text{number of classifications}} \end{align*} It is thus an overall measure over all classes and as we'll shortly see it's not a good measure to tell an ...

27

TL;DR Accuracy is an improper scoring rule. Don't use it. The slightly longer version Actually, accuracy is not even a scoring rule. So asking whether it is (strictly) proper is a category error. The most we can say is that under additional assumptions, accuracy is consistent with a scoring rule that is improper, discontinuous and misleading. (Don't use ...

25

These are not the same thing and they are often used in different contexts. The Dice score is often used to quantify the performance of image segmentation methods. There you annotate some ground truth region in your image and then make an automated algorithm to do it. You validate the algorithm by calculating the Dice score, which is a measure of how similar ...

25

This answer will mostly focus on $R^2$, but most of this logic extends to other metrics such as AUC and so on. This question can almost certainly not be answered well for you by readers at CrossValidated. There is no context-free way to decide whether model metrics such as $R^2$ are good or not. At the extremes, it is usually possible to get a consensus ...

23

You should ask yourself what were you trying to achieve with your modeling approach. As you correctly said "how far from true solution am I" is a good starting point (notice this is also true for classification, we only get into specifics when we run into dichotomization, usually in more CS oriented machine learning, such as trees or SVMs). So, let's ...

21

In the linked blog post, Rob Hyndman calls for entries to a tourism forecasting competition. Essentially, the blog post serves to draw attention to the relevant IJF article, an ungated version of which is linked to in the blog post. The benchmarks you refer to - 1.38 for monthly, 1.43 for quarterly and 2.28 for yearly data - were apparently arrived at as ...

21

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared ...

21

You are getting 100% accuracy because you are using a part of training data for testing. At the time of training, decision tree gained the knowledge about that data, and now if you give same data to predict it will give exactly same value. That's why decision tree producing correct results every time. For any machine learning problem, training and test ...

21

I'll cheat. Specifically, I have argued often (e.g., here) that the statistical part of modeling and prediction extends only to making probabilistic predictions for class memberships (or giving predictive densities, in the case of numerical forecasting). Treating a specific instance as if it belonged to a specific class (or point predictions in the ...

20

F1 score is applicable for any particular point of the ROC curve. This point may represent for example a particular threshold value in a binary classifier and thus corresponds to a particular value of precision and recall. Remember, F score is a smart way to represent both recall and precision. For F score to be high, both precision and recall should be ...

19

Precision can be estimated directly from your data points, but accuracy is related to the experimental design. Suppose I want to find the average height of American males. Given a sample of heights, I can estimate my precision. If my sample is taken from all basketball players, however, my estimate will be biased and inaccurate, and this inaccuracy cannot be ...

19

As other users have told you, you are using as test set a subset of the train set, and a decision tree is very prone to overfitting. You almost had it when you imported from sklearn.cross_validation import train_test_split But then you don't use the function. You should have done: x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33) ...

18

Personally I remember the difference between precision and recall (a.k.a. sensitivity) by thinking about information retrieval: Recall is the fraction of the documents that are relevant to the query that are successfully retrieved, hence its name (in English recall = the action of remembering something). Precision is the fraction of the documents retrieved ...

18

I guess I'm one of the "among others", so I'll chime in. The short version: I'm afraid your example is a bit of a straw man, and I don't think we can learn a lot from it. In the first case, yes, you can threshold your predictions at 0.50 to get a perfect classification. True. But we also see that your model is actually rather poor. Take item #127 in the ...

17

Mnemonics neatly eliminate man’s only nemesis: insufficient cerebral storage. There is SNOUT SPIN: A Sensitive test, when Negative rules OUT disease A Specific test, when Positive, rules IN a disease. I imagine a pig spinning around in a centrifuge, perhaps in preparation for going into space, to help me remember this mnemonic. Humming the theme to Tail ...

16

Probabilistic forecasts (or, as they are also known, density forecasts) can be evaluated using scoring-rules, i.e., functions that map a density forecast and an observed outcome to a so-called score, which is minimized in expectation if the density forecast indeed is the true density to be forecasted. Proper scoring rules are scoring rules that are minimized ...

15

Imbalanced classes in your dataset To be short: imagine, 99% of one class (say apples) and 1% of another class is in your data set (say bananas). My super duper algorithm gets an astonishing 99% accuracy for this data set, check it out: return "it's an apple" He will be right 99% of the time and therefore gets a 99% accuracy. Can I sell you my algorithm? ...

14

There are many different ways of measuring forecast accuracy, and the accuracy() function from the forecast package for R outputs several of them. From your comment about "% of deviation" it sounds like you want to use Mean Absolute Percentage Error, which is one of the measures provided by accuracy(). The most common measures of forecast accuracy are ...

14

Maybe: beware. When you say that 70% accuracy (however you measure it) is good enough for you, it feels like you're assuming that errors are randomly or evenly distributed. But one of the ways of looking at overfitting is that it happens when a model technique allows (and its training process encourages) paying too much attention to quirks in the training ...

Only top voted, non community-wiki answers of a minimum length are eligible