Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Diebold Mariano test Nested Models

I have computed forecasts with 4 different methods, namely OLS, Elastic Net, Cubic splines in combination with Lasso, and Neural Network. All models use the same set of base variables, except cubic ...
0
votes
0answers
22 views

Can MAE be interpreted as the average standard deviation around the true value of a prediction?

MAE is defined as the Mean Absolute Error, that is how far on average the prediction (derived from some prediction model) lies from the real value. The standard deviation is usually interpreted as ...
0
votes
0answers
10 views

inferential approach for estimating error rate on classified population

I am looking mainly for ideas and approaches which I could not find by just Googling. I created a classification model to predict about 175 unique classes from text features. I trained the model on ...
1
vote
1answer
44 views

Computing the accuracy of an answer

Let's assume I have three experts providing an answer. Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%). Expert 2 is 90% accurate Expert 3 is 85% accurate They are ...
1
vote
1answer
35 views

100% accuracy on training, high accuracy on testing as well. What does this mean?

I was training a model to classify different traffic signs and decided to use a pre-trained alexnet model and redefining the last fully-connected layer to match the classes of the dataset. When I did ...
2
votes
1answer
35 views

How to evaluate the accuracy of a probability distribution?

I've trained a Gaussian Bayesian Network. If I feed input values for the parent variables of my output variable, I get a normal distribution. How can I quantify the accuracy of this distribution when ...
1
vote
0answers
17 views

How can I define a accuracy measure for word2vec predictions

I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
0
votes
0answers
9 views

how to evaluate or fulfill required accuracy for regression aka precision of estimation?

Maybe there is already a question similar to mine but there are so many involving the term accuracy and at least none, except of How to evaluate instrumentation accuracy? , didn't seem "very" similar. ...
0
votes
1answer
42 views

Grouped 7-fold Cross Validation in R

I am searching for a grouped 7-fold cross validation function. I couldn't find it in the caret package. I got 70 subjects performing 7 trials (Outcome variable: categorical with 7 values) = 490 ...
1
vote
1answer
26 views

Why are we interested in top-n accuracy?

I know the definition of top-n accuracy: [1] What is the definition of Top-n accuracy? My question is, why do we even care for this? A short answer could be to compare different models in various ...
1
vote
2answers
76 views

Understanding MASE Value

I've looked through many of the other posts concerning the Mean Absolute Scaled Error (MASE) forecast metric and haven't been able to sort out my problem just yet. I'm working with some weather ...
0
votes
0answers
20 views

How to describe accuracy/error without ground truth?

I am using machine learning regression models to predict motor scores among a population with spinal cord injury using features derived from their actual movements. Although the clinical measure we ...
0
votes
1answer
30 views

Validation Accuracy Increases But Training Accuracy Doesn't

The training accuracy of my model is not improving though validation accuracy improves steadily. This is weird abnormal behaviour and I just can't figure out what's wrong. Here are some graphs to help ...
11
votes
2answers
533 views

Is my model any good, based on the diagnostic metric ($R^2$ / AUC / accuracy / etc) value?

I've fitted my model and am trying to understand whether it's any good. I've calculated the recommended metrics to assess it ($R^2$/ AUC / accuracy / prediction error / etc) but do not know how to ...
0
votes
1answer
14 views

Expected Misclustering rate

I am reading this paper on minimax clustering error rates on high-dimensional Gaussian mixtures. The authors define a metric for expected misclustering rate as follows: For a two-component ...
0
votes
0answers
11 views

Reference Request: Overall and Average Accuracy [duplicate]

I am looking for a source to cite on the definitions of overall and average accuracy. I have found many informal sources online, including here on CVSE, but the papers and textbooks I have found seem ...
0
votes
1answer
10 views

Accuracy and F-mesure for imbalanced datasets

I have 10 imbalanced datasets. Classes are : 1, 2, 3, ..., 10,11,12. I used as evaluation metrics for my datasets accuracy and F-measure. The F-messure of each class in each dataset is as below: Is ...
0
votes
1answer
24 views

Measuring predictive accuracy of an ordinal outcome when the predictor is continuous

I'm predicting a response measured by a 5-point likert scale from a model that produces continuous predictions of the same variable. For example: ...
0
votes
0answers
19 views

what test is suitble to predict seen and unseen data

In my model, I want to predict both seen and unseen data and get the result that maximizes the accuracy. The problem is that in some cases seen data are well predicted since the model is overfitting, ...
1
vote
0answers
20 views

Ideal score of a model on training and cross validation data

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here. I have trained a classifier model for binary classification task. Now ...
0
votes
0answers
14 views

Do I choose the logistic regression model with the better F1 statistic on the validation set or the lower AIC?

I am deciding whether to keep an interaction term in my logistic regression model. If I keep it in, the AIC of the model improves. If I leave it out, I get a better F1 score with my validation set. ...
2
votes
1answer
33 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
1
vote
0answers
11 views

Downsampling, AUROC and accuracy equal

I am using downsampling to create perfectly balanced classes in my target feature. I have found that accuracy is exactly equal to the AUROC score. I was thinking that this is because I've used ...
0
votes
0answers
43 views

Is there a name for “normalized accuracy” as a statistic?

In short: I'm using a statistic representing the "normalized accuracy" of a confusion matrix. Is there a formal term for this? $$\text{normacc} = \frac{\text{acc}-\text{thacc}}{1-\text{thacc}}$$ ...
0
votes
1answer
18 views

Correct calculation of repeated cross-validation classification metrics

We can obtain a resampled estimate of training set classification accuracy from caret::confusionMatrix.train(model) e.g., ...
0
votes
1answer
21 views

Does the threshold value of a logistic regression hypothesis has an effect on the accuracy?

It is true that the threshold value of a logistic regression hypothesis has an effect on the Precision/Recall metrics. Suppose you have trained a logistic regression classifier which is outputting $...
0
votes
0answers
25 views

How to compare Accuracy of two RandomForest models? (Chi Sqr or Cohen's H?)

I've got two dataset which have exactly the same structure (15 features, 1 class variable with 7 categories) and roughly the same amount of observations). I trained a Random Forest with the full ...
1
vote
2answers
52 views

Is the accuracy equal if the true positive and false positive rates are equal between two groups?

I was reading the paper "Equality of Opportunity in Supervised Learning" (link). In that paper there is a feature $A \in \{0,1\}$ and a binary outcome $y \in \{0,1\}$. The population is divided into ...
0
votes
0answers
14 views

What type of accuracy should I reported in research paper?

I have read some research papers on the classification task of deep learning, and now I am doing my own. After investigating some research paper which also provided the source code for reproducing ...
2
votes
1answer
66 views

Can balanced accuracy be higher than accuracy?

I have classification tree where the balanced accuracy of the test set is higher than the normal accuracy. I thought balanced accuracy can only have at his maximum the same value as the accuracy not ...
1
vote
0answers
17 views

How to compare predicted accuracy and actual accuracy? [duplicate]

Consider a classifier that, given an input vector ${\bf x}$ outputs both a prediction $y'$ whose accuracy ($a \in \{0, 1\}$) can be measured, as well as a predicted accuracy that corresponds to the ...
1
vote
0answers
27 views

Why we always get different accuracy for a different number of training our model? [duplicate]

As the question says for example if I train my neural network (with 2 layers) model the first time it gives me a score $A \in \mathbb{R}$ and when I train the same model again it gives a different ...
1
vote
1answer
37 views

Discrimination (pseudo) $R^2$ vs. C-index

In the context of binary logistic regression. Both pseudo $R^2$ and C-index measures the discrimination of the model. But why do you need both ? can you gain something from one but not from the other?
2
votes
1answer
35 views

Constant validation loss and increasing validation accuracy

I am training a fully convolutional network. The loss is decreasing whilst the validation loss stays mostly where it is. There is some variance in the validation loss. I thought it might overfits, ...
0
votes
0answers
23 views

How can the balanced accurcay be bigger than the normal accuracy in unbalanced test data? [duplicate]

I constructed two binary classification tree's on two different training set's that i balanced with oversampling and undersampling. The test set is still unbalanced. After that i computetd the ...
4
votes
1answer
145 views

What is the difference between an accuracy measure and an error metric?

The two concepts are distinct in measure theory. Nonetheless, moving out from measure theory, the two terms are often used interchangeably. To most forecasters, especially forecast practitioners, they ...
0
votes
0answers
12 views

Best practice for presenting classifier accuracy using cross-validation?

I have a classifier whose classification accuracy I must present. Of course, I could just do e.g. a single 90/10 train/test split and report the test accuracy. However, my dataset is fairly small, so ...
0
votes
0answers
32 views

Neural Network Accuracy Bouncing Around and Never Going Over 50% Accuracy [Not Duplicate] [duplicate]

My NN accuracy is bouncing between .29 and .37. Sometimes it starts at .5, but then decreases as it continues. The loss also bounces around, decreasing, increasing, and generally staying around 1. The ...
0
votes
3answers
111 views

Machine learning without test and validation data

All mainstream machine learning approaches I've seen depend on a test and usually a validation dataset to measure model accuracy during and after training. This seems like it uses up quite a lot of ...
0
votes
1answer
34 views

Is it possible to estimate accuracy, precision and recall with the given data?

Background: I talked to my friend today and according to herm(him/her) I can calculate precision, recall and accuracy with the current information. Total instances T: 19,532. Instances belonging to ...
1
vote
1answer
27 views

Error measure for 3D-field-comparison - meaning of mean/median

Short Version I have to compare two vectors of predictions (from different methods) against one vector of measurements to find out which prediction performs better. Note that this is not a ...
1
vote
2answers
181 views

accuracy of a regression prediction model

I developed two prediction models using non-linear regression analysis to predict a set of values using sigmoidal and power functions. I was wondering how I can evaluate the accuracy of these ...
0
votes
1answer
82 views

random forest regression predicts “opposite”

I have a dataset with 70 features, which are continuous measures and are interrelated but not highly correlated ($|\rho| <.5$. I have several outcomes, which are each integer values ranging from 0-...
2
votes
1answer
26 views

Precision vs. Accuracy when talking about MSE

This is more of a semantic question. I'm working on translating a work from French to English related to statistics. In French, there is only 1 word as far as I can tell to describe both bias and ...
1
vote
1answer
46 views

Which value of accuracy or balanced accuracy is enough?

I constructed a classification tree and want validate the out of sample performance. I read that the accuracy or the balanced accuracy must at least higher than the no information rate. By the no ...
0
votes
0answers
23 views

Knn Classification on iris dataset

I'm following along https://rpubs.com/Drmadhu/IRISclassification to understand Knn classification. Here's the code I have: library(FNN) iris.sample<-sample.int(n=nrow(irisdat),size=floor(0.75*...
2
votes
1answer
187 views

Learning a quadratic function using TensorFlow/Keras

Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited. I am trying to train a simple neural network to learn a simple quadratic ...
2
votes
1answer
37 views

Which approach should be used to compare two different measurement techniques of same samples?

I have individually measured failure forces of 8 materials and those recorded with A method and B method in same time: 8 results in each method, A=8 and B=8. The range of data of both measurement ...
1
vote
1answer
13 views

Is it appropriate to use a confusion matrix for frequency data?

I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
0
votes
0answers
91 views

Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...