Questions tagged [accuracy]
Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)
536
questions
0
votes
0answers
9 views
R: Accuracy Sensitivity/Specificity
I want to use Accuracy in a Multiple-Choice Test as my Dependent Variable.
I have questions with 6 possible answers, 1 answer is always correct.
I have 3 columns: Response, Right Response, Accuracy
<...
0
votes
0answers
7 views
Measure for in equality of prediction quality in mutlclass classification
I have got a balanced dataset of 10 different classes $y\in\{0,\dots 9\}$. After fitting a ML model I get classification results $y^*$. Despite the data being balanced, the results are not.
I want a ...
0
votes
0answers
12 views
How to compute the loss function using Mc Fadden pseudo r-squared
I have currently developed an optimised deep learning model (DNN) using cross-entropy loss function. To objectively compare my DNN model with conditional logistics I need to obtain McFadden pseudo r2 ...
1
vote
0answers
29 views
Under which circumstances does LDA achieve a higher classification accuracy than QDA?
Since some weeks, I pursue the question "Under which circumstances will LDA achieve a higher classification accuracy than QDA using the same training and test set as well as the same prior ...
0
votes
0answers
7 views
Generalization performance of a model (RStudio)
I'm kind of confused on the definition of generalization. E.g. I've created a model in RStudio, for which I've plotted the training history results:
Now I'm wondering, is the generalization of my ...
0
votes
1answer
25 views
checking accuracy of a forecast [closed]
I am trying to predict future sale of a product by using holt-winters
imp info:
nextsalesf contains the forecast for next 5 periods
val$Qty.2011001FBL0010250[5] is the known values of next 5 ...
1
vote
2answers
34 views
Reporting F1 Scores
I have a question with regard to the proper way to report F1 scores. Say I am comparing two algorithms one with F1 score of 0.71 and the other of 0.82.
Is it correct to say:
"Algorithm 1 obtained an ...
0
votes
0answers
20 views
Why the Logistic regression model trained with tensorflow performed so poor
I trained a logistic regression model with tensorflow but the accuracy of the model was so poor (accuracy = 0.68). The model was trained using simulated dataset and the result should be very good. is ...
0
votes
0answers
48 views
Text classification - High Accuracy, low recall and low precision
I am using fastai to create a text classifier that labels texts as either 0 or 1.
My data (number of 1's and 0's) for training is balanced, and I got an accuracy of 85%.
To test, I used a new ...
0
votes
0answers
12 views
How to test accuracy manually [duplicate]
I am programing a neural network from scratch in C++ (purely for fun).
My dataset is cut in 3 : 1 train, 1 validation for early stopping and 1 test to measure model accuracy.
I measure accuracy ...
0
votes
1answer
27 views
R caret classification - why doesn't model accuracy equal accuracy given by predict()?
I have a dataset with 1000 samples, and each sample is 1 of 3 classes. I'm training classifiers on the dataset and predicting classes (5-fold cross-validated) and I'd like to know how well each ...
1
vote
1answer
18 views
What is the difference between the average class accuracy and average instance accuracy?
In some papers, I find some strange evaluation metrics that did not give a clear explanation, like average class accuracy and average instance accuracy!
What is the difference between both? I can not ...
1
vote
1answer
22 views
Good metric to assess error in estimating a value
So this seems like a simple question, but I cant find a way to solve it or formulate a solution that makes sense.
My case is that I have an algo that detects fuel theft (ft_calc). Now I want to ...
2
votes
1answer
36 views
Very basic question about task
I'm sorry if the question is too basic but I really couldn't decide where to ask it
As a machine learning super beginner I decided to play around with some programs that can perform action ...
1
vote
0answers
35 views
Understanding Accuracy, Recall and IoU
Working on an image segmenetation problem, I've tackled the following scenario repeated on different images:
High Recall and Accuracy (around 99%) Low IoU (around 60%)
How is that possible?
Recall ...
0
votes
0answers
36 views
Diebold Mariano test Nested Models
I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic ...
0
votes
0answers
31 views
Can MAE be interpreted as the average standard deviation around the true value of a prediction?
MAE is defined as the Mean Absolute Error, that is how far on average the prediction (derived from some prediction model) lies from the real value.
The standard deviation is usually interpreted as ...
0
votes
0answers
11 views
inferential approach for estimating error rate on classified population
I am looking mainly for ideas and approaches which I could not find by just Googling.
I created a classification model to predict about 175 unique classes from text features. I trained the model on ...
1
vote
1answer
46 views
Computing the accuracy of an answer
Let's assume I have three experts providing an answer.
Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%).
Expert 2 is 90% accurate
Expert 3 is 85% accurate
They are ...
1
vote
1answer
79 views
100% accuracy on training, high accuracy on testing as well. What does this mean?
I was training a model to classify different traffic signs and decided to use a pre-trained alexnet model and redefining the last fully-connected layer to match the classes of the dataset. When I did ...
2
votes
1answer
39 views
How to evaluate the accuracy of a probability distribution?
I've trained a Gaussian Bayesian Network. If I feed input values for the parent variables of my output variable, I get a normal distribution. How can I quantify the accuracy of this distribution when ...
1
vote
0answers
24 views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
0
votes
0answers
12 views
how to evaluate or fulfill required accuracy for regression aka precision of estimation?
Maybe there is already a question similar to mine but there are so many involving the term accuracy and at least none, except of How to evaluate instrumentation accuracy? , didn't seem "very" similar.
...
0
votes
1answer
121 views
Grouped 7-fold Cross Validation in R
I am searching for a grouped 7-fold cross validation function. I couldn't find it in the caret package.
I got 70 subjects performing 7 trials (Outcome variable: categorical with 7 values) = 490 ...
1
vote
1answer
39 views
Why are we interested in top-n accuracy?
I know the definition of top-n accuracy: [1] What is the definition of Top-n accuracy?
My question is, why do we even care for this?
A short answer could be to compare different models in various ...
1
vote
2answers
92 views
Understanding MASE Value
I've looked through many of the other posts concerning the Mean Absolute Scaled Error (MASE) forecast metric and haven't been able to sort out my problem just yet.
I'm working with some weather ...
0
votes
0answers
32 views
How to describe accuracy/error without ground truth?
I am using machine learning regression models to predict motor scores among a population with spinal cord injury using features derived from their actual movements. Although the clinical measure we ...
0
votes
1answer
34 views
Validation Accuracy Increases But Training Accuracy Doesn't
The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help ...
11
votes
3answers
1k views
Is my model any good, based on the diagnostic metric ($R^2$/ AUC/ accuracy/ RMSE etc.) value?
I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
0
votes
1answer
16 views
Expected Misclustering rate
I am reading this paper on minimax clustering error rates on high-dimensional Gaussian mixtures. The authors define a metric for expected misclustering rate as follows:
For a two-component ...
0
votes
0answers
14 views
Reference Request: Overall and Average Accuracy [duplicate]
I am looking for a source to cite on the definitions of overall and average accuracy. I have found many informal sources online, including here on CVSE, but the papers and textbooks I have found seem ...
0
votes
1answer
12 views
Accuracy and F-mesure for imbalanced datasets
I have 10 imbalanced datasets. Classes are : 1, 2, 3, ..., 10,11,12.
I used as evaluation metrics for my datasets accuracy and F-measure.
The F-messure of each class in each dataset is as below:
Is ...
0
votes
1answer
32 views
Measuring predictive accuracy of an ordinal outcome when the predictor is continuous
I'm predicting a response measured by a 5-point likert scale from a model that produces continuous predictions of the same variable.
For example:
...
0
votes
0answers
19 views
what test is suitble to predict seen and unseen data
In my model, I want to predict both seen and unseen data and get the result that maximizes the accuracy. The problem is that in some cases seen data are well predicted since the model is overfitting, ...
1
vote
0answers
22 views
Ideal score of a model on training and cross validation data
The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.
I have trained a classifier model for binary classification task. Now ...
0
votes
0answers
14 views
Do I choose the logistic regression model with the better F1 statistic on the validation set or the lower AIC?
I am deciding whether to keep an interaction term in my logistic regression model. If I keep it in, the AIC of the model improves. If I leave it out, I get a better F1 score with my validation set. ...
2
votes
1answer
35 views
Balanced datasets are almost all predicted negative
Problem
I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
1
vote
0answers
12 views
Downsampling, AUROC and accuracy equal
I am using downsampling to create perfectly balanced classes in my target feature. I have found that accuracy is exactly equal to the AUROC score. I was thinking that this is because I've used ...
0
votes
0answers
66 views
Is there a name for “normalized accuracy” as a statistic?
In short: I'm using a statistic representing the "normalized accuracy" of a confusion matrix. Is there a formal term for this?
$$\text{normacc} = \frac{\text{acc}-\text{thacc}}{1-\text{thacc}}$$
...
0
votes
1answer
23 views
Correct calculation of repeated cross-validation classification metrics
We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model)
e.g.,
...
0
votes
1answer
26 views
Does the threshold value of a logistic regression hypothesis has an effect on the accuracy?
It is true that the threshold value of a logistic regression hypothesis has an effect on the Precision/Recall metrics.
Suppose you have trained a logistic regression classifier which is outputting $...
0
votes
0answers
25 views
How to compare Accuracy of two RandomForest models? (Chi Sqr or Cohen's H?)
I've got two dataset which have exactly the same structure (15 features, 1 class variable with 7 categories) and roughly the same amount of observations). I trained a Random Forest with the full ...
1
vote
2answers
54 views
Is the accuracy equal if the true positive and false positive rates are equal between two groups?
I was reading the paper "Equality of Opportunity in Supervised Learning" (link).
In that paper there is a feature $A \in \{0,1\}$ and a binary outcome $y \in \{0,1\}$. The population is divided into ...
0
votes
0answers
16 views
What type of accuracy should I reported in research paper?
I have read some research papers on the classification task of deep learning, and now I am doing my own. After investigating some research paper which also provided the source code for reproducing ...
2
votes
1answer
97 views
Can balanced accuracy be higher than accuracy?
I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
1
vote
0answers
20 views
How to compare predicted accuracy and actual accuracy? [duplicate]
Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
1
vote
0answers
30 views
Why we always get different accuracy for a different number of training our model? [duplicate]
As the question says for example if I train my neural network (with 2 layers) model the first time it gives me a score $A \in \mathbb{R}$ and when I train the same model again it gives a different ...
1
vote
1answer
54 views
Discrimination (pseudo) $R^2$ vs. C-index
In the context of binary logistic regression. Both pseudo $R^2$ and C-index measures the discrimination of the model. But why do you need both ? can you gain something from one but not from the other?
2
votes
1answer
41 views
Constant validation loss and increasing validation accuracy
I am training a fully convolutional network. The loss is decreasing whilst the validation loss stays mostly where it is. There is some variance in the validation loss.
I thought it might overfits, ...
0
votes
0answers
23 views
How can the balanced accurcay be bigger than the normal accuracy in unbalanced test data? [duplicate]
I constructed two binary classification tree's on two different training set's that i balanced with oversampling and undersampling. The test set is still unbalanced.
After that i computetd the ...
