Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

164 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
127 views

Question 10.9 from Bayesian Data Analysis, what does accuracy mean here?

I'm doing an independent study in Bayesian Statistics following some chapters from BDA3. When solving the first question from Ch 10 I got stuck. It says: [If] a scalar variable $\theta$ is ...
4
votes
0answers
230 views

Can F1-Score be higher than accuracy?

I'm using sklearn's confusion_matrix and classification_report methods to compute the ...
4
votes
1answer
172 views

Accuracy of a polygon fitting algorithm

I have a polygon $P$ which I have to fit with a rectangle $R$, as shown in the figure below. This is done by varying the dimension of the rectangle, and choosing the dimension that best fits the ...
2
votes
1answer
32 views

Precision vs. Accuracy when talking about MSE

This is more of a semantic question. I'm working on translating a work from French to English related to statistics. In French, there is only 1 word as far as I can tell to describe both bias and ...
2
votes
1answer
39 views

Which approach should be used to compare two different measurement techniques of same samples?

I have individually measured failure forces of 8 materials and those recorded with A method and B method in same time: 8 results in each method, A=8 and B=8. The range of data of both measurement ...
2
votes
1answer
39 views

With test accuracy being equal, is it better to have lower training accuracy?

Suppose we train two models on a training set, and then test them both on the training set itself, and on a test set. We have some accuracy metric we're using to evaluate them. Both models score ...
2
votes
0answers
352 views

Calculation of accuracy (and Cohen's kappa) using sensitivity, specificity, positive and negative predictive values

I read How to calculate specificity from accuracy and sensitivity, but I have two diagnostic performance measures more. Please correct me if I am wrong: if Sensitivity=TP/(TP+FN) Specificity=TN/(TN+...
2
votes
1answer
128 views

Classification model accuracy, roc auc score, f1 score 100%

I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't ...
2
votes
0answers
303 views

Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below). All ...
2
votes
0answers
196 views

Difference between cumulative gains chart and cumulative accuracy profile for binary classifier

I am confused about the following: Here I find the definition of cumulative gains chart as the plot of x-axis: the rate of predicted positive y-axis: true positive rate It turns out that we can e....
2
votes
0answers
101 views

Binary Classification: good at predicting negative class but bad at predicting positive class

I used many different algorithms on a data set for binary classification. I used kNN, SVM radial, ANN, random forest, Gaussian process classification, etc. Every algorithm is well tuned in R using ...
2
votes
0answers
290 views

Is it always better to use F1-Score than Accuracy as performance metric?

During reading papers about Machine Learning I always find researchers using accuracy as their sole performance metric. However, a high accuracy alone proves nothing when the amount of false positives ...
2
votes
0answers
118 views

Comparing two split criteria for decision trees

Consider a split criterion for decision trees, which favors splits resulting in groups with as evenly distributed classes as possible. What will be the effect on the resulting decision trees compared ...
2
votes
0answers
114 views

Does It make sense to interprete variables with low accuracy but high R squared in Random Forest?

I did random forest analysis and I obtained two graphs. In the first one (left hand side) we can see the measure of importance of each predictor and on the second one (right hand side) is the ...
2
votes
0answers
202 views

Rleation between L2 loss and accuracy

Given a classification problem with k classes, suppose a model outputting a probability distribution over the classes that uses some gradient-based learning method is used (yes, in my case it's neural ...
2
votes
1answer
77 views

Classifying users in one of two groups

I've a - probably beginner level - question about classifying. My goal is to classify users as UG1 or UG2, based on some specific characteristics of them. So, what I've done so far is: I have two ...
2
votes
0answers
145 views

How would you interpret decreasing cost but increasing training and validation error during epochs?

I’m training a fully convolution network with final layer of global sum pooling and no intermediate pooling in the network which I already test on global average pooling and it converged very well. ...
2
votes
0answers
129 views

How to compare the forecast accuracy of two models when the data has unbalanced panel structure?

I am comparing the earnings forecast accuracy of two models (model 1 and model 2). The data (firm-year level data) have an unbalanced panel structure since firms have varying lifetime and time to be ...
2
votes
0answers
3k views

How to threshold multiclass probability prediction to get confusion matrix?

Lets say my multinomial logistic regression predict that a chance of a sample belonging to a each class is A=0.6, B=0.3, C=0.1 How do I threshold this values to get just binary prediction of a sample ...
2
votes
0answers
168 views

Calculating NPV from precision, recall and CA

I am trying to simulate predictions of a binary classifier, given its precision, recall, classification accuracy ($CA$) and the actual classes of the instances. I.e. I have instances and I know what ...
2
votes
0answers
89 views

Variable Importance's reliability when model accuracy is low

I am trying to predict the termination events for a company. I also want to rank the most important variables that lead to termination. I ran a decision tree model and have the following questions: ...
2
votes
0answers
72 views

biological sample size when instrument accuracy is weak?

I'm using a porometer to measure the transpiration of grape leaves. Unfortunately, the instrument's accuracy is 10% and it has a relatively slow (60 seconds) round trip time to produce readings when ...
2
votes
0answers
745 views

Is using Rpart with unbalanced data a good idea?

I have a rather unbalanced data set and want to use rpart to build a classification tree. After building the full tree, I prune it back using the 1-SE rule. On average, only 1-2 splits are suggested. ...
2
votes
0answers
314 views

Interpreting Elastic Net Coefficients and Accuracy Testing

I have a code for an elastic net model which is predicting the best lambda to use. With this, it provides coefficients for those values. I wanted to see how accurate those coefficients are that it ...
2
votes
0answers
1k views

Test MAPE < Train MAPE using auto.arima()

I am trying to build a forecasting model for the passenger vehicles registrations in a given country, and I wanted to use $auto.arima$ function from the $forecast$ package to estimate a simple ARIMA ...
2
votes
0answers
74 views

How do you report percent error with limited precision?

Let's say you have an experimental data point with limited precision, such as 0.67 ± 0.1, and a known actual value such as 0.72. What is the standard way to report the percent error in this case, ...
2
votes
0answers
585 views

Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
2
votes
0answers
853 views

How to calculate accuracy of each feature

I read some paper (* about predicting user retention in StumbledUpon) and saw the authors provide a list of features with accuracy of each feature with the following explanation: As decision ...
2
votes
1answer
62 views

How do you combine the error of a device with the standard deviation of the readings from it?

I have recorded 3 readings of pressure at a set flow rate for about 8 different flow rates. I know the accuracy of my manometer, and can calculate stddev from the 3 points for each. How do you combine ...
1
vote
0answers
29 views

Understanding Accuracy, Recall and IoU

Working on an image segmenetation problem, I've tackled the following scenario repeated on different images: High Recall and Accuracy (around 99%) Low IoU (around 60%) How is that possible? Recall ...
1
vote
0answers
21 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
0answers
21 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
1
vote
0answers
12 views

Downsampling, AUROC and accuracy equal

I am using downsampling to create perfectly balanced classes in my target feature. I have found that accuracy is exactly equal to the AUROC score. I was thinking that this is because I've used ...
1
vote
0answers
10 views

What measures of an ML algorithm's 'accuracy' are mostly consistent as the number of classes to predict into varies?

For a research project, I've got a bunch (N=507) of 20-second VR tracking data clips (6DOF x head and hands), each from a different participant. My goal is to predict the participant using a small ...
1
vote
0answers
91 views

Measure of accuracy for a Bayesian model

I am reading Statistical Rethinking (Section 6.2.1.2). The topic of this section is measuring accuracy for a Bayesian model, i.e. accuracy of the model of predicting correctly an outcome. The ...
1
vote
0answers
101 views

Naive Bayes and kNN accuracy

Assume that a large number of binary features are added to a dataset with two class labels c1 and c2, such that for each added feature f, the class conditional probability P(f = 0|c1) = P(f = 0|c2). ...
1
vote
0answers
42 views

How to know if your decision tree model has overfitting or not?

I am using the DecisionTreeClassifier() of python and I am changing some tuning parameters to understand if my model has overfitting or no because when I exectue the following code without any ...
1
vote
0answers
53 views

How to correctly compare the accuracy of different forecasting methods using bootstrapping with time series forecasting

I am currently working on a forecasting project and I have tried several different models to forecast with. Having trained and tuned my models I want to pick which model works best for each time ...
1
vote
0answers
31 views

Boosted decision trees: in which situations are “deep” decision trees performing better?

The general idea of boosted decision trees is to use very simple trees in the following manner (simplified, for intuition only): start with a simple tree, fit another simple tree on the residuals, ...
1
vote
0answers
37 views

Compare accuracy between tools using k-fold cross validation, each tool is tested with different k values

I'm working on a new way to do the classification in a supervised way and I want to compare its accuracy to some related works. These works are using the same data set and they are testing their ...
1
vote
0answers
164 views

Confusions about Pesaran & Timmermann test (2009 version)

I am learning about Pesaran Timmermann test (2009) for directional accuracy and I have troubles understanding its formula. Here I use notation and arrangement used in Pönkä (2017): $$PT=(T-1)(S^{-1}_{...
1
vote
0answers
54 views

Metric to evaluate accuracy of time series decomposition?

I am trying to de-seasonalize a large number of business sales data. They all come from the same industry and follow idiosyncratic seasonality patterns. The number of businesses is quite large, and I ...
1
vote
0answers
34 views

How to make really bad results from a machine learning model better by reversing predictions

I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
1
vote
0answers
128 views

Alternative measurement for Cross Entropy Loss than accuracy when using soft labeling?

Suppose one has multi-class ($n>2$) data, where class labels are soft, e.g. two samples of for 3-classed data might look like: ...
1
vote
0answers
429 views

Recall equals to accuracy but different to precision

I've read this question, and basically I'm having the same issue. I'm dealing with a binary classification problem. I'm calculating the precision, recall and f1 using ...
1
vote
0answers
100 views

Can i use Diebold Mariano test for comparison of 2 models across multiple time series?

I have 2 models (for simplicity, let's call them AR(1) and MA(1)) each making 1 day ahead forecasts of time series. If I had only 1 time series I would just use ...
1
vote
0answers
115 views

Comparing two models with statistical testing

I am working on a neural network architecture to tackle a problem and I want to compare my model to another model used in literature. I use k-fold crossvalidation to get a more unbiased accuracy and ...
1
vote
0answers
57 views

Can we increase the accuracy of a classifier using sketches?

I am using a sketch technique to improve the memory of a standard classifier (naive Bayes) with data streams. The sketch technique is composed of a sketch table (hash table) means the true values can ...
1
vote
1answer
207 views

Statistical hypothesis test for classification accuracy

I just want to make sure I'm doing things right! I created 2 algorithms that classify my data into 2 groups. The first one gives me an accuracy of 76.66% (sensitivity=0.76, specificity=0.78). The ...
1
vote
1answer
58 views

Measures of accuracy for discrete variables

I am testing the accuracy of discrete variable prediction (>= 2 possible outcomes). I've seen things like using a confusion matrix or ROC curve for binary outcomes, but not much for > 2 outcome ...