# Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

69 questions
Filter by
Sorted by
Tagged with
7answers
33k views

### Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
2answers
579 views

### Is my model any good, based on the diagnostic metric ($R^2$ / AUC / accuracy / etc) value?

I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
1answer
16k views

### What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

The Mean Absolute Percentage Error (mape) is a common accuracy or error measure for time series or other predictions, $$\text{MAPE} = \frac{100}{n}\sum_{t=1}^n\frac{|A_t-F_t|}{A_t}\%,$$ where $A_t$ ...
1answer
980 views

### Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
4answers
34k views

### Training a decision tree against unbalanced data

I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy. The data consists of students ...
2answers
19k views

### Interpretation of mean absolute scaled error (MASE)

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006). $$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$ where $MAE$ is the mean absolute error ...
1answer
1k views

### Why not using the R squared to measure forecast accuracy?

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)? I was thinking about the difference: By using ...
1answer
985 views

### Why does the accuracy not change, when applying different alpha values in L2 regularisation

Figure below shows the accuracy using different alpha values in L2 regularisation. As long as alpha is small in the range of $10^{-12}$ to $10^{-2}$ the accuracy remain the same. I do undarstand when ...
2answers
260 views

### The dirty coins and the three judges

All things being perfect, if we want to know if a set of coinage (all coins with the same bias) is biased, and to what degree, we would just use the outcomes and confidence intervals of the binomial ...
5answers
5k views

### Is an overfitted model necessarily useless?

Assume that a model has 100% accuracy on the training data, but 70% accuracy on the test data. Is the following argument true about this model? It is obvious that this is an overfitted model. The ...
1answer
15k views

### Good accuracy despite high loss value

During the training of a simple neural network binary classifier I get an high loss value, using cross-entropy. Despite this, accuracy's value on validation set holds quite good. Does it have some ...
2answers
618 views

### When are OLS linear regression parameters inaccurate?

Q1: Show quantitatively that OLS regression can be applied inconsistently for linear parameters estimation. We show an example of linear OLS inaccuracy from inappropriate application to bivariate ...
1answer
1k views

### Stacking: Do more base classifiers always improve accuracy?

When using stacking, can accuracy always be improved by adding more base classifiers, types of base classifiers, and features?
1answer
1k views

### Evaluate forecasting ability of GARCH models with RMSE and MAE

I am evaluating different forecasting models and their ability to forecast index volatility during period of market turmoil, using two measurements, Root Mean Square Error and Mean Absolute Error. For ...
2answers
5k views

### Logistic regression - how good is my model? [duplicate]

I am a beginner in ML so apologize in advance if this sounds silly. I did a logistic regression on a real data set and I am having problems measuring how well my model fits. I still don't understand ...
1answer
540 views

### Alternative to MAPE when the data is not a time series

I have a data set where many of the actual values are zero, so I can't use MAPE. It's not a time series, so I can't use MASE ala our very own Rob Hyndman. Is there another alternative to MAPE that I ...
1answer
7k views

### Diebold-Mariano test for predictive accuracy

I am using the Diebold-Mariano test in the forecast package of R to test the predictive accuracy. In particular, I want to ...
1answer
1k views

### Why is feature selection important, for classification tasks?

I'm learning about feature selection. I can see why it would be important and useful, for model-building. But let's focus on supervised learning (classification) tasks. Why is feature selection ...
2answers
649 views

### Example when using accuracy as an outcome measure will lead to a wrong conclusion

I am looking into various performance measures for predictive models. A lot was written about problems of using accuracy, instead of something more continuous to evaluate model performance. Frank ...
1answer
355 views

### Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
1answer
1k views

### Diebold-Mariano test for multiple prediction horizons

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$. I deduced from this post ...
2answers
670 views

### How to predict using ordered probit regression and calculate prediction accuracy?

I want to do an ordered probit regression, then cross-validate model prediction accuracy with 80% data for training and 20% for validation, and calculate RMSE for predictions. Consider this dataset: ...
2answers
3k views

### Difference between forecasting accuracy and forecasting error?

I am working on a demand forecasting project and I am puzzled by the client's standards of forecast evaluation. The MAPE (Mean Absolute Percentage Error) with the sample data Forecast = 300 and Demand ...
1answer
75 views

### When is accuracy score preferred to AUCROC?

I have a binary and balanced dataset. Do I have to see the AUROC as the different trade-offs between the TPR and the FPR and the accuracy as a result with a threshold of 0.5? When is accuracy a ...
1answer
3k views

### Difference between Mean/average accuracy and Overall accuracy

I just got confusion while reading the paper "Local Binary Pattern-Based Hyperspectral Image Classification With Superpixel Guidance". They mentioned that they repeated each experiment 10 times and ...
6answers
9k views

### What is the best way to remember the difference between sensitivity, specificity, precision, accuracy, and recall?

Despite having seen these terms 502847894789 times, I cannot for the life of me remember the difference between sensitivity, specificity, precision, accuracy, and recall. They're pretty simple ...
1answer
12k views

### F1/Dice-Score vs IoU

I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar ...
3answers
14k views

### What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
1answer
291 views

### Voting system that uses accuracy of each voter and the associated uncertainty

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they ...
4answers
19k views

### Forecast accuracy calculation

We are using STL (R implementation) for forecasting time series data. Every day we run daily forecasts. We would like to compare forecast values with real values and identify average deviation. For ...
1answer
3k views

### Why the sum of true positive and false positive does not have to be equal to one?

While reading this question above, I got confused about the sum of true positive and false positive. If an aircraft is present in a certain area, a radar detects it and generates an alarm signal ...
1answer
10k views

### How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
1answer
371 views

### Should predictive accuracy or, alternatively, minimizing the MSE, be reconsidered?

Ever since Breiman, maximizing predictive accuracy has become a predictive modeling gold standard, of sorts. That it has evolved to this status is understandable: it can be "optimized," is easily ...
1answer
438 views

### How to decide the “best” accuracy score for prediction of binary outcome?

Dr Frank Harrell mentioned in his book and BIOS 330 course that Accuracy score used to drive model building should be a continuous score that utilizes all the information in the data (e.g. Brier ...
4answers
4k views

### Statistical Significance of multiple classifiers by using p-value

I had a classification problem. I had 675 samples, and used with 7 machine learning algorithms with 10 cross-validation for prediction. Lets say the following table is the accuracy result of each ...
2answers
1k views

### Classification accuracy based on probability

Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
3answers
392 views

### Accuracy of a probability estimate

How can you classify the accuracy of a probability? Say I do a study of people that like bananas in 2 different regions. Region 1: 8 out of 10 people tested like bananas. Region 2: 500 out of 1000 ...
1answer
138 views

### logistic regression predictive modeling

I would like to use a logistic regression for estimating the parameters of the logit function by using the maximum likelihood estimate. This amounts to minimizing the log-loss function, instead of ...
2answers
10k views

### Is f-measure synonymous with accuracy?

I understand that f-measure (based on precision and recall) is an estimate of how accurate a classifier is. Also, f-measure is favored over accuracy when we have an unbalanced dataset. I have a simple ...
1answer
2k views

### What's the measure to assess the binary classification accuracy for imbalanced data?

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
2answers
4k views

### Completely different results after each cross validation

I'm running some classification algorithms in MATLAB and validating them with a 10-fold cross validation. The problem is that every time I execute the cross validation, it gives a very different ...
3answers
799 views

### Optimize a classification algorithm using mean per-class accuracy

I have a binary classification problem and am trying to find a way to optimize my machine learning algorithm using a performance metric based on the per-class error rate. If I'm not misinterpreting ...
1answer
168 views

### Accuracy of point forecasts vs. average accuracy of multistep forecasts?

It seems to me that it is possible that a forecasting model does very well on one step ahead forecasts (or on any other point forecast) but performs poorly on multistep forecasts (if you average the ...
2answers
5k views

### What does it imply if accuracy and recall are the same?

I did a number of machine learning experiments to predict a binary classification. I measured precision, recall and accuracy. I noticed that my precision is generally quite high, and recall and ...
1answer
2k views

### How to calculate specificity from accuracy and sensitivity

By having accuracy and sensitivity, can we calculate specificity? How about calculating sensitivity from accuracy and specificity? My second question is, if sensitivity and specificity both ...
2answers
162 views

### Choosing between logistic and discriminant

I am looking at regularized logistic regression, (l1 and l2 at the moment) and regularized discriminant analysis. How do I compare the two? I was thinking of doing gcv on both methods over a set of ...
2answers
2k views

### AUC vs accuracy for model accuracy evaluation [duplicate]

In R I tried to measure the accuracy by performing a classification analysis using logistic regression analysis. I found that there are two ways to measure accuracy. One is AUC measurement using ROC ...
1answer
1k views

### Diebold-Mariano test in case of nested models (Clark & McCracken, 2001)

I have become aware of Clark & McCracken (2001) showing that the asymptotics of the Diebold-Mariano test will potentially collapse when comparing forecast accuracy of nested models (such as GARCH /...
2answers
3k views

### Prediction Accuracy - Another Measurement than MAPE

I have the question for my prediction model, which estimates used car prices. For example: Car: 20.000 km, real price: 32.000 Euro, prediction: 27.000 Euro --> MAPE: 15,6% , absolute difference : ...
2answers
16k views

### Is accuracy = 1- test error rate

Apologies if this is a very obvious question, but I have been reading various posts and can't seem to find a good confirmation. In the case of classification, is a classifier's accuracy = 1- test ...