Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

Filter by
Sorted by
Tagged with
113
votes
7answers
40k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
50
votes
6answers
10k views

What is the best way to remember the difference between sensitivity, specificity, precision, accuracy, and recall?

Despite having seen these terms 502847894789 times, I cannot for the life of me remember the difference between sensitivity, specificity, precision, accuracy, and recall. They're pretty simple ...
43
votes
4answers
37k views

Training a decision tree against unbalanced data

I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy. The data consists of students ...
38
votes
6answers
17k views

Why do I get a 100% accuracy decision tree?

I'm getting a 100% accuracy for my decision tree. What am I doing wrong? This is my code: ...
29
votes
1answer
19k views

What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

The Mean Absolute Percentage Error (mape) is a common accuracy or error measure for time series or other predictions, $$ \text{MAPE} = \frac{100}{n}\sum_{t=1}^n\frac{|A_t-F_t|}{A_t}\%,$$ where $A_t$ ...
25
votes
5answers
6k views

Is an overfitted model necessarily useless?

Assume that a model has 100% accuracy on the training data, but 70% accuracy on the test data. Is the following argument true about this model? It is obvious that this is an overfitted model. The ...
24
votes
1answer
16k views

F1/Dice-Score vs IoU

I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar ...
22
votes
2answers
21k views

Interpretation of mean absolute scaled error (MASE)

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006). $$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$ where $MAE$ is the mean absolute error ...
19
votes
3answers
1k views

How can we judge the accuracy of Nate Silver's predictions?

Firstly, he gives probability of outcomes. So, for example, his predictions for the U.S. election is currently 82% Clinton vs 18% Trump. Now, even if Trump wins, how do I know that it wasn't just the ...
18
votes
1answer
12k views

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
15
votes
3answers
2k views

Are there parameters where a biased estimator is considered “better” than the unbiased estimator? [duplicate]

A perfect estimator would be accurate (unbiased) and precise (good estimation even with small samples). I never really thought of the question of precision but only the one of accuracy (as I did in ...
15
votes
2answers
3k views

If “Standard error” and “Confidence intervals” measure precision of measurement, then what are the measurements of accuracy?

In book "Biostatistics for dummies" in page 40 I read: The standard error (abbreviated SE) is one way to indicate how precise your estimate or measurement of something is. and Confidence ...
15
votes
2answers
17k views

Good accuracy despite high loss value

During the training of a simple neural network binary classifier I get an high loss value, using cross-entropy. Despite this, accuracy's value on validation set holds quite good. Does it have some ...
15
votes
2answers
6k views

Accuracy vs. area under the ROC curve

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
14
votes
3answers
15k views

What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
13
votes
2answers
19k views

Is accuracy = 1- test error rate

Apologies if this is a very obvious question, but I have been reading various posts and can't seem to find a good confirmation. In the case of classification, is a classifier's accuracy = 1- test ...
12
votes
2answers
27k views

Is the Dice coefficient the same as accuracy?

I come across the Dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). ...
12
votes
3answers
2k views

Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?

I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
12
votes
1answer
1k views

Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
12
votes
1answer
12k views

How to determine the accuracy of regression? Which measure should be used?

I have problem with defining the unit of accuracy in a regression task. In classification tasks is easy to calculate sensitivity or specificity of classifier because output is always binary {correct ...
11
votes
1answer
1k views

Why is feature selection important, for classification tasks?

I'm learning about feature selection. I can see why it would be important and useful, for model-building. But let's focus on supervised learning (classification) tasks. Why is feature selection ...
11
votes
1answer
295 views

Voting system that uses accuracy of each voter and the associated uncertainty

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they ...
10
votes
2answers
11k views

Is f-measure synonymous with accuracy?

I understand that f-measure (based on precision and recall) is an estimate of how accurate a classifier is. Also, f-measure is favored over accuracy when we have an unbalanced dataset. I have a simple ...
10
votes
1answer
3k views

Evaluation of classifiers: learning curves vs ROC curves

I would like to compare 2 different classifiers for a multiclass text classification problem that use large training datasets. I am doubting whether I should use ROC curves or learning curves to ...
9
votes
4answers
20k views

Forecast accuracy calculation

We are using STL (R implementation) for forecasting time series data. Every day we run daily forecasts. We would like to compare forecast values with real values and identify average deviation. For ...
9
votes
4answers
38k views

Interpretation of AIC value

Typical values of AIC that I have seen for logistic models are in thousands, at least hundreds. e.g. On http://www.r-bloggers.com/how-to-perform-a-logistic-regression-in-r/ the AIC is 727.39 While ...
8
votes
2answers
874 views

Example when using accuracy as an outcome measure will lead to a wrong conclusion

I am looking into various performance measures for predictive models. A lot was written about problems of using accuracy, instead of something more continuous to evaluate model performance. Frank ...
8
votes
1answer
62 views

How to dissuade laypeople from drawing inaccurate conclusions about their data?

I work as a data analyst, primarily in SQL, providing operations data to internal customers. I rarely do statistical analysis. Recently, internal customers have been coming to me with data from ...
7
votes
1answer
2k views

Why not using the R squared to measure forecast accuracy?

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)? I was thinking about the difference: By using ...
7
votes
1answer
372 views

Should predictive accuracy or, alternatively, minimizing the MSE, be reconsidered?

Ever since Breiman, maximizing predictive accuracy has become a predictive modeling gold standard, of sorts. That it has evolved to this status is understandable: it can be "optimized," is easily ...
7
votes
3answers
223 views

How to know if a statistic prediction is right?

The weather forecast predict the probability of rain or not for some day. If I could repeat the same day many times I could count how many times does it rain or not, so I could compare with the ...
6
votes
1answer
10k views

Time series forecasting accuracy measures: MAPE and MASE

We come to this toy example showing MAPE and MASE are not consistent when measuring forecasting accuracy. Data consist of 100 white noise and 100 $AR(1)$ time series with length $N=500$, mean $\mu=1$ ...
6
votes
1answer
3k views

Why the sum of true positive and false positive does not have to be equal to one?

While reading this question above, I got confused about the sum of true positive and false positive. If an aircraft is present in a certain area, a radar detects it and generates an alarm signal ...
6
votes
4answers
5k views

Should I get 100% classification accuracy on training data?

I've been getting inconsistent results with a binary classification problem I'm trying to solve using a linear classifier and a custom feature extraction pipeline, and decided to do a quick check of ...
6
votes
1answer
2k views

What's the measure to assess the binary classification accuracy for imbalanced data?

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
6
votes
1answer
25k views

Calculating MAPE [closed]

Is the below calculation of the Mean Absolute Percentage Error MAPE correct? I've included a workable example, but really the lines in question are these: ...
6
votes
3answers
4k views

Accuracy of model lower than no-information rate?

I have a dataset with predicted variable having two classes; true and false. 99.99% of the values are with false class. In this case, no-information rate is 99.99%. So, any model that I build needs to ...
6
votes
1answer
7k views

Diebold-Mariano test for predictive accuracy

I am using the Diebold-Mariano test in the forecast package of R to test the predictive accuracy. In particular, I want to ...
6
votes
1answer
397 views

Is the objective to beat a random classifier when the data set is skewed using PR curves?

I have a testing data set where 1/3 of the observations are class-1 objects and the remainder class-0. Hence, the data set is skewed (skewed classifier), literature suggests that if the data set is ...
6
votes
1answer
218 views

Should OOB (Out Of Bag) error be less than a Test set error in Random Forests?

I am using the book, "An introduction to statistical learning with applications in R" and reading the section on using OOB to estimate the model error for Random Forests. The graph seems to suggest ...
5
votes
4answers
2k views

Loss vs. Classification Accuracy in applied problems

In practical problems, where we want to for instance predict if a subject has a certain disease or not, we usually take classification accuracy as a measure which has the straightforward ...
5
votes
3answers
2k views

Forecast accuracy metric that involves prediction intervals

I'm in the process of generating a time series forecast for a company's product revenue and am looking for some way to show accuracy over time - e.g. after say 6 months they want to see how the actual ...
5
votes
2answers
398 views

How to select the best classifier in classification task?

Suppose in the task of classification, I want to select the best classifier. How to do this? My idea is to select the classifier which gives the highest classification accuracy using the test data ...
5
votes
1answer
9k views

AUC and Balanced accuracy in R Modelling

Can someone please explain the difference between AUC(Area under curve) and balanced accuracy in R? For eg: In decision tree modelling I got the, AUC : 0.91 balanced accuracy : 0.72 please explain ...
5
votes
1answer
605 views

How to transform an accuracy distribution for a violin plot

I am trying to find the best way to visualize different distributions of accuracy. Accuracy here is a value in the interval [0,1], 0 meaning not accurate, 1 maximum accuracy. I have different methods ...
5
votes
2answers
3k views

validation/training accuracy and overfitting

If we randomly split the data into training data and validation data, and assume the training data and validation data have similar "distributions", i.e. they are both good representations of the ...
5
votes
1answer
3k views

Evaluating relative accuracy/error for continuous value prediction (and assessing relative average difference/error)

I was trying to understand what are good (or standard) way of evaluating relative accuracy for continuous data. Say for example, say I have some statistical algorithm S that outputs some real number ...
5
votes
3answers
116 views

How do I call a forecast that is both accurate and precise?

How do I call a forecast (more precisely, a forecasting rule) that is both accurate and precise? Is there a word that expresses both properties combined? I do not mean the forecasting rule is perfect,...
5
votes
2answers
938 views

Which is the best method for meta-analysis of diagnostic test accuracy studies?

I am conducting a meta-analysis of diagnostic test accuracy studies focusing on myocardial perfusion imaging. I have used first Meta-Disc, but only for descriptive purposes, as it is clear that ...
5
votes
0answers
127 views

Question 10.9 from Bayesian Data Analysis, what does accuracy mean here?

I'm doing an independent study in Bayesian Statistics following some chapters from BDA3. When solving the first question from Ch 10 I got stuck. It says: [If] a scalar variable $\theta$ is ...