Questions tagged [accuracy]
Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)
776
questions
0
votes
0
answers
13
views
Why PR score is down when balanced accuracy is good?
I just read this discussion here and here.
I have a dataset of 977 records where class proportion is 77:23.
My balanced accuracy is 75.5, ...
- 1,780
0
votes
2
answers
29
views
Comparing impact of training data size - what testing data size?
I am training a classifier using BERT and want to check how the accuracy changes with increasing training data size. Up until now, I have 1k annotated training samples and tested the accuracy for ...
- 3
0
votes
1
answer
22
views
What metrics work well in unbalanced assemblies?
I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
- 125
1
vote
0
answers
16
views
Validation accuracy does not reflect the actual change in the testing accuracy
When training a CNN classifier for the Fashion-MNIST, I have noticed that there are multiple instances that validation error is not improving even if the tesing accuracy is improving every epoch or ...
- 287
1
vote
0
answers
20
views
Training loss goes back up but validation accuracy continues growing (XGBoost)
Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba.
Main ...
- 519
0
votes
0
answers
24
views
"Predictive dependence" between two variables
Given two random variables $X$ and $Y$, it is natural to use the conditional entropy $H[Y|X]$ to quantify the extent to which knowing $X$ decreases the uncertainty about $Y$. However, consider the ...
- 247
1
vote
1
answer
21
views
Logistic regression predictions dont work
I have this problem with logit, that when I want to create confusion matrix, it simply displays the real values in the first row and in the second row, there are never any numbers. I created a lot of ...
- 11
0
votes
0
answers
9
views
How to increase accuracy and to receive better forecasts?
I am building a LSTM network and I am using the following structure:
...
- 11
1
vote
1
answer
22
views
Accuracy failure example
I am trying to understand the functioning of accuracy and I need a practical example.
This is what I understood: it gets the average correctness of the predictions and in some cases its result can be ...
- 145
0
votes
0
answers
27
views
Accuracy for unfairness detection
I am dealing with unfairness and I am trying to find out some metrics to detect unfairness presence in my dataset. I am starting from the very bases: accuracy. My main question is Do you think I can ...
- 145
0
votes
0
answers
17
views
What does it mean if the validation accuracy is equal to the testing accuracy?
I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
- 13
2
votes
1
answer
37
views
Will threshold of logistic regression change accuracy? Any relationship with the incidence of disease? [duplicate]
I am using a logistic regression model to predict breast cancer. I trained and tested the model in a population with a pretty high incidence of breast cancer(since the individuals all went to the ...
- 461
1
vote
1
answer
18
views
Which best model I have to use to calculate standard deviations?
Suppose that I run the same experiment 5 times.
Now, suppose that each model of each experiment has the best accuracy in different epochs. For example:
Model1: best accuracy after 10 epochs
Model2: ...
- 23
1
vote
0
answers
62
views
Why did cross validation accuracy suddenly decrease? and then increase again?
I am training a classification model in Tensorflow. Here is a screenshot of the graphs I obtained after training.
The blue one is the training accuracy, while the orange one is the cross validation ...
- 111
0
votes
0
answers
6
views
Interpreting validation loss and accuracy for various learning rates
I am having a hard time comparing the effect of different learning rates on validation loss and accuracy. Would I be right in assuming that a Learning rate of 0.0001 was the most successful as the ...
- 1
0
votes
1
answer
18
views
Understanding the idx column in h2o metrics [closed]
h2o model metrics results report generated like this.
...
- 3
0
votes
1
answer
36
views
How to evaluate te testing data from the trained data?
I am new at ML, and still trying to understand some concepts, so I figured I could ask here and maybe finally understand.
How does it work the whole splitting data from a data set?
Before answering, ...
1
vote
1
answer
30
views
Why is a 3-copula better than a 2-copula?
Suppose that I have known that $X$ and $Y$ have high dependency, $Y$ and $Z$ have high dependency, and $Z$ and $X$ also have high dependency through three different 2-copulas.
Suppose I fit one 3-...
- 135
0
votes
0
answers
10
views
Metric to estimate measuring accuracy
I need help/suggestions on which metric/approach to use.
I am trying to estimate pointing precision by measuring the offset of the experimentally measured values from the real, theoretical values, ...
- 101
2
votes
2
answers
65
views
A/B test with a result of another deep learning analysis with 90% accuracy
I am planning to conduct a A/B test with data obtained through a deep learning algorithm. Say, I got a binary classification dataset through machine learning with about 100k rows classified into yes, ...
- 53
6
votes
2
answers
193
views
Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss
Vladimir Vapnik wrote:
“When solving a problem of interest, do not solve a more general
problem as an intermediate step. Try to get the answer that you really
need but not a more general one.”
...
- 48.5k
0
votes
1
answer
40
views
Compare model accuracy with null accuracy
My current model of Logistic Regression return a accuracy of:
Training set score: 0.8476
Test set score: 0.8502
Comparing model accuracy with null accuracy:
...
1
vote
0
answers
22
views
Which Proper Scoring rule for when? [duplicate]
Hi, I'm quite new to statistics and have been tasked to evaluate if there is a difference in accuracy between 2 subpopulations in a logistic model.
The credit scoring company's model calculates the ...
- 11
0
votes
0
answers
15
views
Assess results of non-linear continous model
I have a set of y_true labels and a set of y_pred labels. The prediction model is actually a deep learning model in combination with a rule-based information extractor. It tries to find the ...
- 1
0
votes
0
answers
29
views
Determine test accuracy
I'm trying to analyze results of a certain study. In this study, a certain diagnostic test was conducted in order to see if the pre-test assumed diagnosis was correct. Management was changed in about ...
0
votes
0
answers
17
views
What metric can be considered to find if causal impact has determined the best combination of synthetic controls
I'm new to Causal impact. I read the paper and video by Kay which has a detailed description of the package.
Can someone suggest any metric which can describe the accuracy of the synthetic control ...
- 1
0
votes
1
answer
25
views
How do I obtain a percentage accuracy for an LSTM
I've got an LSTM trained for time series forecasting and I've seen people online report their LSTMs accuracy in %s such as 85% accurate etc, how do I obtain a metric like this? so far I was just using ...
1
vote
0
answers
23
views
multi class prediction metric
I would like a method that generalized metrics for multiclass. Including imbalanced classes.
This metric should be all against all.
I thought of Generalizing the Mathew's coefficient but not in the ...
- 11
0
votes
0
answers
73
views
Explain MAAPE (Mean Arctangent Absolute Percentage Error) in simple terms (intermittent demand forecasting)
In order to measure the accuracy of highly intermitted demand time series, I recently discovered a new accuracy measure, that overcomes the problem of zero values and values close to zero, when ...
- 101
0
votes
0
answers
12
views
Regression vs Classification at Accuracy
Let's assume we want to predict some continuous values within y_regression in [-100,100], and all the X (train set) is continuous.
Now, let's say we care only about ...
- 171
1
vote
1
answer
29
views
Is Average Accuracy An Ill-Defined Metric? [closed]
Five people each took a 100 question yes/no question test. I have their individual accuracy scores and was planning to report their average.
However, I was informed that "You cannot take an ...
- 33
0
votes
0
answers
8
views
How can I find the percentage accuracy between an approximation and a source formula?
Im trying to find the percentage of error for a trigonometric sine graph.
I wrote an approximation of the graph, and am comparing the two up close, https://www.shadertoy.com/view/NslfRs
Red represents ...
- 101
0
votes
0
answers
23
views
Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?
I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
- 3
1
vote
0
answers
34
views
the value of accuracy and loss change by the order of the training
https://colab.research.google.com/drive/13wMNCXxKs_uqFVzuJqE2zxhsxWK8Ya4k?usp=sharing
Hello guys
I’m having a hard time trying to figure out what I am doing wrong here.
I used 4 pretrained models from ...
we ing
1
vote
1
answer
82
views
Balanced accuracy reduces to accuracy for balanced datasets
This question might be trivial, but I have problems understanding this line taken from here:
The balanced_accuracy_score function computes the balanced accuracy, which avoids inflated performance ...
- 113
1
vote
1
answer
118
views
calculating the top k accuracy using logits vs softmax probabilites
I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
- 11
1
vote
0
answers
15
views
What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]
As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting.
Consider a case of ...
0
votes
0
answers
23
views
Comparing classification metrics among studies
I am doing some comparison of classification results obtained by various studies using machine learning. All studies use the same "type" of data (satellite imagery) but differ in many ...
- 21
1
vote
2
answers
155
views
Reporting the best test accuracy in a research paper?
I am running a random forest classifier for binary classification. I have split the dataset into training and testing. From training data, I select 70% of the data for hyperparameter tuning, I used 5 ...
- 139
2
votes
0
answers
25
views
Training Set: Why Loss Flatten but Accuracy continues to increase?
I took a the Coursera course: Convolutional Neural Networks in TensorFlow, and one of the quiz questions is
When exploring the graphs, the loss levelled out at about .75 after 2 epochs, but the ...
- 131
1
vote
0
answers
17
views
Comparing models with partially observed data and limited supervision
Setup: Suppose we have $n$ questions and for each question $m$ answers. For each question, a model, Model A, selects one answer out of $m$. Note that for each question, zero, one, or more answers ...
- 670
0
votes
1
answer
54
views
Accuracy, Recall (sensitivity), Specificity confusion
I came up with this paper with 765 citations!
On page 2, it expresses an equation relating these metrics:
Accuracy
sensitivity (a.k.a Recall).
specificity (a.k.a 1 - type1 error rate)
prevalence (...
- 325
0
votes
1
answer
66
views
Does it matter if real data will be imbalanced, if the ML model was trained on a balanced dataset?
I have trained a machine learning model (supervised, classification, LinearSVC) on a balanced dataset, which produces relatively good results on the test data. I am happy with the numbers, but not ...
- 3
1
vote
0
answers
31
views
Underfitting, overfitting or good fit?
I am working with a medical dataset, trying to figure out people with diabetes. I implemented LSTM since the target variable is a categorical one, the accuracy gave %84. I plotted the accuracy ...
- 11
0
votes
0
answers
24
views
Is it appropriate to use difference scores in this context?
I am trying to compare the level of rating accuracy of two groups of participants (say Sample A and B). The study design is as is: I asked a consumer panel to rate how much they liked three products. ...
- 1
0
votes
0
answers
82
views
Can you calculate recall/precision/f1 score for "continuous variables"
My goal is to calculate some performance metrics (precision, recall, f1,accuracy) on a binary variable (yes or no) but have dates associated with each data point. Attaching dates to this binary ...
- 1
0
votes
0
answers
74
views
How to increase the accuracy of a multivariate LSTM network?
I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
- 11
0
votes
0
answers
14
views
What social/economic/political domains are typ. easier-harder to forecast ? Are there benchmark estimates of accuracy for such time series forecasts?
For the report of the results from the Behavioral and Social Science Forecasting Collaborative Tournament I organized during the first COVID year, our team is trying to identify good benchmarks for ...
1
vote
1
answer
57
views
ARIMA accuracy measures, rolling forecast
Regarding ARIMA model selection and especially accuracy measures several questions came into my mind. To shortly summarize, in my understanding, after necessary transformations/differencing, p and q ...
- 59
1
vote
0
answers
16
views
how to increase location accuracy?
I am developing a website for student attendance if a student is present in classroom in given time and he/she fill attendance by pressing a button then his/her attendance should be marked as present.
...
- 11