Questions tagged [accuracy]
Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)
523
questions
0
votes
0answers
19 views
Diebold Mariano test Nested Models
I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic ...
0
votes
0answers
22 views
Can MAE be interpreted as the average standard deviation around the true value of a prediction?
MAE is defined as the Mean Absolute Error, that is how far on average the prediction (derived from some prediction model) lies from the real value.
The standard deviation is usually interpreted as ...
0
votes
0answers
10 views
inferential approach for estimating error rate on classified population
I am looking mainly for ideas and approaches which I could not find by just Googling.
I created a classification model to predict about 175 unique classes from text features. I trained the model on ...
1
vote
1answer
44 views
Computing the accuracy of an answer
Let's assume I have three experts providing an answer.
Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%).
Expert 2 is 90% accurate
Expert 3 is 85% accurate
They are ...
1
vote
1answer
35 views
100% accuracy on training, high accuracy on testing as well. What does this mean?
I was training a model to classify different traffic signs and decided to use a pre-trained alexnet model and redefining the last fully-connected layer to match the classes of the dataset. When I did ...
2
votes
1answer
35 views
How to evaluate the accuracy of a probability distribution?
I've trained a Gaussian Bayesian Network. If I feed input values for the parent variables of my output variable, I get a normal distribution. How can I quantify the accuracy of this distribution when ...
1
vote
0answers
17 views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
0
votes
0answers
9 views
how to evaluate or fulfill required accuracy for regression aka precision of estimation?
Maybe there is already a question similar to mine but there are so many involving the term accuracy and at least none, except of How to evaluate instrumentation accuracy? , didn't seem "very" similar.
...
0
votes
1answer
42 views
Grouped 7-fold Cross Validation in R
I am searching for a grouped 7-fold cross validation function. I couldn't find it in the caret package.
I got 70 subjects performing 7 trials (Outcome variable: categorical with 7 values) = 490 ...
1
vote
1answer
26 views
Why are we interested in top-n accuracy?
I know the definition of top-n accuracy: [1] What is the definition of Top-n accuracy?
My question is, why do we even care for this?
A short answer could be to compare different models in various ...
1
vote
2answers
76 views
Understanding MASE Value
I've looked through many of the other posts concerning the Mean Absolute Scaled Error (MASE) forecast metric and haven't been able to sort out my problem just yet.
I'm working with some weather ...
0
votes
0answers
20 views
How to describe accuracy/error without ground truth?
I am using machine learning regression models to predict motor scores among a population with spinal cord injury using features derived from their actual movements. Although the clinical measure we ...
0
votes
1answer
30 views
Validation Accuracy Increases But Training Accuracy Doesn't
The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help ...
11
votes
2answers
533 views
Is my model any good, based on the diagnostic metric ($R^2$ / AUC / accuracy / etc) value?
I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
0
votes
1answer
14 views
Expected Misclustering rate
I am reading this paper on minimax clustering error rates on high-dimensional Gaussian mixtures. The authors define a metric for expected misclustering rate as follows:
For a two-component ...
0
votes
0answers
11 views
Reference Request: Overall and Average Accuracy [duplicate]
I am looking for a source to cite on the definitions of overall and average accuracy. I have found many informal sources online, including here on CVSE, but the papers and textbooks I have found seem ...
0
votes
1answer
10 views
Accuracy and F-mesure for imbalanced datasets
I have 10 imbalanced datasets. Classes are : 1, 2, 3, ..., 10,11,12.
I used as evaluation metrics for my datasets accuracy and F-measure.
The F-messure of each class in each dataset is as below:
Is ...
0
votes
1answer
24 views
Measuring predictive accuracy of an ordinal outcome when the predictor is continuous
I'm predicting a response measured by a 5-point likert scale from a model that produces continuous predictions of the same variable.
For example:
...
0
votes
0answers
19 views
what test is suitble to predict seen and unseen data
In my model, I want to predict both seen and unseen data and get the result that maximizes the accuracy. The problem is that in some cases seen data are well predicted since the model is overfitting, ...
1
vote
0answers
20 views
Ideal score of a model on training and cross validation data
The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.
I have trained a classifier model for binary classification task. Now ...
0
votes
0answers
14 views
Do I choose the logistic regression model with the better F1 statistic on the validation set or the lower AIC?
I am deciding whether to keep an interaction term in my logistic regression model. If I keep it in, the AIC of the model improves. If I leave it out, I get a better F1 score with my validation set. ...
2
votes
1answer
33 views
Balanced datasets are almost all predicted negative
Problem
I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
1
vote
0answers
11 views
Downsampling, AUROC and accuracy equal
I am using downsampling to create perfectly balanced classes in my target feature. I have found that accuracy is exactly equal to the AUROC score. I was thinking that this is because I've used ...
0
votes
0answers
43 views
Is there a name for “normalized accuracy” as a statistic?
In short: I'm using a statistic representing the "normalized accuracy" of a confusion matrix. Is there a formal term for this?
$$\text{normacc} = \frac{\text{acc}-\text{thacc}}{1-\text{thacc}}$$
...
0
votes
1answer
18 views
Correct calculation of repeated cross-validation classification metrics
We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model)
e.g.,
...
0
votes
1answer
21 views
Does the threshold value of a logistic regression hypothesis has an effect on the accuracy?
It is true that the threshold value of a logistic regression hypothesis has an effect on the Precision/Recall metrics.
Suppose you have trained a logistic regression classifier which is outputting $...
0
votes
0answers
25 views
How to compare Accuracy of two RandomForest models? (Chi Sqr or Cohen's H?)
I've got two dataset which have exactly the same structure (15 features, 1 class variable with 7 categories) and roughly the same amount of observations). I trained a Random Forest with the full ...
1
vote
2answers
52 views
Is the accuracy equal if the true positive and false positive rates are equal between two groups?
I was reading the paper "Equality of Opportunity in Supervised Learning" (link).
In that paper there is a feature $A \in \{0,1\}$ and a binary outcome $y \in \{0,1\}$. The population is divided into ...
0
votes
0answers
14 views
What type of accuracy should I reported in research paper?
I have read some research papers on the classification task of deep learning, and now I am doing my own. After investigating some research paper which also provided the source code for reproducing ...
2
votes
1answer
66 views
Can balanced accuracy be higher than accuracy?
I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
1
vote
0answers
17 views
How to compare predicted accuracy and actual accuracy? [duplicate]
Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
1
vote
0answers
27 views
Why we always get different accuracy for a different number of training our model? [duplicate]
As the question says for example if I train my neural network (with 2 layers) model the first time it gives me a score $A \in \mathbb{R}$ and when I train the same model again it gives a different ...
1
vote
1answer
37 views
Discrimination (pseudo) $R^2$ vs. C-index
In the context of binary logistic regression. Both pseudo $R^2$ and C-index measures the discrimination of the model. But why do you need both ? can you gain something from one but not from the other?
2
votes
1answer
35 views
Constant validation loss and increasing validation accuracy
I am training a fully convolutional network. The loss is decreasing whilst the validation loss stays mostly where it is. There is some variance in the validation loss.
I thought it might overfits, ...
0
votes
0answers
23 views
How can the balanced accurcay be bigger than the normal accuracy in unbalanced test data? [duplicate]
I constructed two binary classification tree's on two different training set's that i balanced with oversampling and undersampling. The test set is still unbalanced.
After that i computetd the ...
4
votes
1answer
145 views
What is the difference between an accuracy measure and an error metric?
The two concepts are distinct in measure theory. Nonetheless, moving out from measure theory, the two terms are often used interchangeably. To most forecasters, especially forecast practitioners, they ...
0
votes
0answers
12 views
Best practice for presenting classifier accuracy using cross-validation?
I have a classifier whose classification accuracy I must present. Of course, I could just do e.g. a single 90/10 train/test split and report the test accuracy. However, my dataset is fairly small, so ...
0
votes
0answers
32 views
Neural Network Accuracy Bouncing Around and Never Going Over 50% Accuracy [Not Duplicate] [duplicate]
My NN accuracy is bouncing between .29 and .37. Sometimes it starts at .5, but then decreases as it continues. The loss also bounces around, decreasing, increasing, and generally staying around 1. The ...
0
votes
3answers
111 views
Machine learning without test and validation data
All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training.
This seems like it uses up quite a lot of ...
0
votes
1answer
34 views
Is it possible to estimate accuracy, precision and recall with the given data?
Background: I talked to my friend today and according to herm(him/her) I can calculate precision, recall and accuracy with the current information.
Total instances T: 19,532.
Instances belonging to ...
1
vote
1answer
27 views
Error measure for 3D-field-comparison - meaning of mean/median
Short Version
I have to compare two vectors of predictions (from different methods) against one vector of measurements to find out which prediction performs better. Note that this is not a ...
1
vote
2answers
181 views
accuracy of a regression prediction model
I developed two prediction models using non-linear regression analysis to predict a set of values using sigmoidal and power functions. I was wondering how I can evaluate the accuracy of these ...
0
votes
1answer
82 views
random forest regression predicts “opposite”
I have a dataset with 70 features, which are continuous measures and are interrelated but not highly correlated ($|\rho| <.5$. I have several outcomes, which are each integer values ranging from 0-...
2
votes
1answer
26 views
Precision vs. Accuracy when talking about MSE
This is more of a semantic question. I'm working on translating a work from French to English related to statistics. In French, there is only 1 word as far as I can tell to describe both bias and ...
1
vote
1answer
46 views
Which value of accuracy or balanced accuracy is enough?
I constructed a classification tree and want validate the out of sample performance. I read that the accuracy or the balanced accuracy must at least higher than the no information rate. By the no ...
0
votes
0answers
23 views
Knn Classification on iris dataset
I'm following along https://rpubs.com/Drmadhu/IRISclassification to understand Knn classification.
Here's the code I have:
library(FNN)
iris.sample<-sample.int(n=nrow(irisdat),size=floor(0.75*...
2
votes
1answer
187 views
Learning a quadratic function using TensorFlow/Keras
Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited.
I am trying to train a simple neural network to learn a simple quadratic ...
2
votes
1answer
37 views
Which approach should be used to compare two different measurement techniques of same samples?
I have individually measured failure forces of 8 materials and those recorded with A method and B method in same time: 8 results in each method, A=8 and B=8. The range of data of both measurement ...
1
vote
1answer
13 views
Is it appropriate to use a confusion matrix for frequency data?
I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
0
votes
0answers
91 views
Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]
I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
