Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

Filter by
Sorted by
Tagged with
100
votes
7answers
33k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
11
votes
2answers
579 views

Is my model any good, based on the diagnostic metric ($R^2$ / AUC / accuracy / etc) value?

I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
27
votes
1answer
16k views

What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?

The Mean Absolute Percentage Error (mape) is a common accuracy or error measure for time series or other predictions, $$ \text{MAPE} = \frac{100}{n}\sum_{t=1}^n\frac{|A_t-F_t|}{A_t}\%,$$ where $A_t$ ...
11
votes
1answer
980 views

Is accuracy an improper scoring rule in a binary classification setting?

I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
40
votes
4answers
34k views

Training a decision tree against unbalanced data

I'm new to data mining and I'm trying to train a decision tree against a data set which is highly unbalanced. However, I'm having problems with poor predictive accuracy. The data consists of students ...
19
votes
2answers
19k views

Interpretation of mean absolute scaled error (MASE)

Mean absolute scaled error (MASE) is a measure of forecast accuracy proposed by Koehler & Hyndman (2006). $$MASE=\frac{MAE}{MAE_{in-sample, \, naive}}$$ where $MAE$ is the mean absolute error ...
7
votes
1answer
1k views

Why not using the R squared to measure forecast accuracy?

Why in literature usually the common accuracy measures like MAD, MSE, RMSE, MAPE ... are used. Why not using the $R^2$ (coefficient of determination)? I was thinking about the difference: By using ...
2
votes
1answer
985 views

Why does the accuracy not change, when applying different alpha values in L2 regularisation

Figure below shows the accuracy using different alpha values in L2 regularisation. As long as alpha is small in the range of $10^{-12}$ to $10^{-2}$ the accuracy remain the same. I do undarstand when ...
4
votes
2answers
260 views

The dirty coins and the three judges

All things being perfect, if we want to know if a set of coinage (all coins with the same bias) is biased, and to what degree, we would just use the outcomes and confidence intervals of the binomial ...
24
votes
5answers
5k views

Is an overfitted model necessarily useless?

Assume that a model has 100% accuracy on the training data, but 70% accuracy on the test data. Is the following argument true about this model? It is obvious that this is an overfitted model. The ...
15
votes
1answer
15k views

Good accuracy despite high loss value

During the training of a simple neural network binary classifier I get an high loss value, using cross-entropy. Despite this, accuracy's value on validation set holds quite good. Does it have some ...
1
vote
2answers
618 views

When are OLS linear regression parameters inaccurate?

Q1: Show quantitatively that OLS regression can be applied inconsistently for linear parameters estimation. We show an example of linear OLS inaccuracy from inappropriate application to bivariate ...
4
votes
1answer
1k views

Stacking: Do more base classifiers always improve accuracy?

When using stacking, can accuracy always be improved by adding more base classifiers, types of base classifiers, and features?
0
votes
1answer
1k views

Evaluate forecasting ability of GARCH models with RMSE and MAE

I am evaluating different forecasting models and their ability to forecast index volatility during period of market turmoil, using two measurements, Root Mean Square Error and Mean Absolute Error. For ...
1
vote
2answers
5k views

Logistic regression - how good is my model? [duplicate]

I am a beginner in ML so apologize in advance if this sounds silly. I did a logistic regression on a real data set and I am having problems measuring how well my model fits. I still don't understand ...
0
votes
1answer
540 views

Alternative to MAPE when the data is not a time series

I have a data set where many of the actual values are zero, so I can't use MAPE. It's not a time series, so I can't use MASE ala our very own Rob Hyndman. Is there another alternative to MAPE that I ...
6
votes
1answer
7k views

Diebold-Mariano test for predictive accuracy

I am using the Diebold-Mariano test in the forecast package of R to test the predictive accuracy. In particular, I want to ...
11
votes
1answer
1k views

Why is feature selection important, for classification tasks?

I'm learning about feature selection. I can see why it would be important and useful, for model-building. But let's focus on supervised learning (classification) tasks. Why is feature selection ...
8
votes
2answers
649 views

Example when using accuracy as an outcome measure will lead to a wrong conclusion

I am looking into various performance measures for predictive models. A lot was written about problems of using accuracy, instead of something more continuous to evaluate model performance. Frank ...
2
votes
1answer
355 views

Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
2
votes
1answer
1k views

Diebold-Mariano test for multiple prediction horizons

I am trying to compare two forecasts using the Mariano Diebold test in R. Both forecasts are for 150 days ahead; that is, on day $t$ I forecast $t+1, t+2, \dotsc, t+150$. I deduced from this post ...
0
votes
2answers
670 views

How to predict using ordered probit regression and calculate prediction accuracy?

I want to do an ordered probit regression, then cross-validate model prediction accuracy with 80% data for training and 20% for validation, and calculate RMSE for predictions. Consider this dataset: ...
3
votes
2answers
3k views

Difference between forecasting accuracy and forecasting error?

I am working on a demand forecasting project and I am puzzled by the client's standards of forecast evaluation. The MAPE (Mean Absolute Percentage Error) with the sample data Forecast = 300 and Demand ...
2
votes
1answer
75 views

When is accuracy score preferred to AUCROC?

I have a binary and balanced dataset. Do I have to see the AUROC as the different trade-offs between the TPR and the FPR and the accuracy as a result with a threshold of 0.5? When is accuracy a ...
1
vote
1answer
3k views

Difference between Mean/average accuracy and Overall accuracy

I just got confusion while reading the paper "Local Binary Pattern-Based Hyperspectral Image Classification With Superpixel Guidance". They mentioned that they repeated each experiment 10 times and ...
46
votes
6answers
9k views

What is the best way to remember the difference between sensitivity, specificity, precision, accuracy, and recall?

Despite having seen these terms 502847894789 times, I cannot for the life of me remember the difference between sensitivity, specificity, precision, accuracy, and recall. They're pretty simple ...
19
votes
1answer
12k views

F1/Dice-Score vs IoU

I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar ...
13
votes
3answers
14k views

What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
11
votes
1answer
291 views

Voting system that uses accuracy of each voter and the associated uncertainty

Let's say, we have simple "yes/no" question that we want to know answer to. And there are N people "voting" for correct answer. Every voter has a history - list of 1's and 0's, showing whether they ...
9
votes
4answers
19k views

Forecast accuracy calculation

We are using STL (R implementation) for forecasting time series data. Every day we run daily forecasts. We would like to compare forecast values with real values and identify average deviation. For ...
6
votes
1answer
3k views

Why the sum of true positive and false positive does not have to be equal to one?

While reading this question above, I got confused about the sum of true positive and false positive. If an aircraft is present in a certain area, a radar detects it and generates an alarm signal ...
17
votes
1answer
10k views

How is the confusion matrix reported from K-fold cross-validation?

Suppose I do K-fold cross-validation with K=10 folds. There will be one confusion matrix for each fold. When reporting the results, should I calculate what is the average confusion matrix, or just sum ...
7
votes
1answer
371 views

Should predictive accuracy or, alternatively, minimizing the MSE, be reconsidered?

Ever since Breiman, maximizing predictive accuracy has become a predictive modeling gold standard, of sorts. That it has evolved to this status is understandable: it can be "optimized," is easily ...
4
votes
1answer
438 views

How to decide the “best” accuracy score for prediction of binary outcome?

Dr Frank Harrell mentioned in his book and BIOS 330 course that Accuracy score used to drive model building should be a continuous score that utilizes all the information in the data (e.g. Brier ...
3
votes
4answers
4k views

Statistical Significance of multiple classifiers by using p-value

I had a classification problem. I had 675 samples, and used with 7 machine learning algorithms with 10 cross-validation for prediction. Lets say the following table is the accuracy result of each ...
3
votes
2answers
1k views

Classification accuracy based on probability

Let's say we have a simple binary classification problem. So for a predictor X we want to predict response Y. Y is binary, so either 0 or 1. Now let's say we use two different classifiers, model1 and ...
2
votes
3answers
392 views

Accuracy of a probability estimate

How can you classify the accuracy of a probability? Say I do a study of people that like bananas in 2 different regions. Region 1: 8 out of 10 people tested like bananas. Region 2: 500 out of 1000 ...
1
vote
1answer
138 views

logistic regression predictive modeling

I would like to use a logistic regression for estimating the parameters of the logit function by using the maximum likelihood estimate. This amounts to minimizing the log-loss function, instead of ...
9
votes
2answers
10k views

Is f-measure synonymous with accuracy?

I understand that f-measure (based on precision and recall) is an estimate of how accurate a classifier is. Also, f-measure is favored over accuracy when we have an unbalanced dataset. I have a simple ...
6
votes
1answer
2k views

What's the measure to assess the binary classification accuracy for imbalanced data?

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
4
votes
2answers
4k views

Completely different results after each cross validation

I'm running some classification algorithms in MATLAB and validating them with a 10-fold cross validation. The problem is that every time I execute the cross validation, it gives a very different ...
4
votes
3answers
799 views

Optimize a classification algorithm using mean per-class accuracy

I have a binary classification problem and am trying to find a way to optimize my machine learning algorithm using a performance metric based on the per-class error rate. If I'm not misinterpreting ...
4
votes
1answer
168 views

Accuracy of point forecasts vs. average accuracy of multistep forecasts?

It seems to me that it is possible that a forecasting model does very well on one step ahead forecasts (or on any other point forecast) but performs poorly on multistep forecasts (if you average the ...
3
votes
2answers
5k views

What does it imply if accuracy and recall are the same?

I did a number of machine learning experiments to predict a binary classification. I measured precision, recall and accuracy. I noticed that my precision is generally quite high, and recall and ...
1
vote
1answer
2k views

How to calculate specificity from accuracy and sensitivity

By having accuracy and sensitivity, can we calculate specificity? How about calculating sensitivity from accuracy and specificity? My second question is, if sensitivity and specificity both ...
1
vote
2answers
162 views

Choosing between logistic and discriminant

I am looking at regularized logistic regression, (l1 and l2 at the moment) and regularized discriminant analysis. How do I compare the two? I was thinking of doing gcv on both methods over a set of ...
0
votes
2answers
2k views

AUC vs accuracy for model accuracy evaluation [duplicate]

In R I tried to measure the accuracy by performing a classification analysis using logistic regression analysis. I found that there are two ways to measure accuracy. One is AUC measurement using ROC ...
0
votes
1answer
1k views

Diebold-Mariano test in case of nested models (Clark & McCracken, 2001)

I have become aware of Clark & McCracken (2001) showing that the asymptotics of the Diebold-Mariano test will potentially collapse when comparing forecast accuracy of nested models (such as GARCH /...
0
votes
2answers
3k views

Prediction Accuracy - Another Measurement than MAPE

I have the question for my prediction model, which estimates used car prices. For example: Car: 20.000 km, real price: 32.000 Euro, prediction: 27.000 Euro --> MAPE: 15,6% , absolute difference : ...
12
votes
2answers
16k views

Is accuracy = 1- test error rate

Apologies if this is a very obvious question, but I have been reading various posts and can't seem to find a good confirmation. In the case of classification, is a classifier's accuracy = 1- test ...