Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Why PR score is down when balanced accuracy is good?

I just read this discussion here and here. I have a dataset of 977 records where class proportion is 77:23. My balanced accuracy is 75.5, ...
user avatar
  • 1,780
0 votes
2 answers
29 views

Comparing impact of training data size - what testing data size?

I am training a classifier using BERT and want to check how the accuracy changes with increasing training data size. Up until now, I have 1k annotated training samples and tested the accuracy for ...
user avatar
  • 3
0 votes
1 answer
22 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
user avatar
1 vote
0 answers
16 views

Validation accuracy does not reflect the actual change in the testing accuracy

When training a CNN classifier for the Fashion-MNIST, I have noticed that there are multiple instances that validation error is not improving even if the tesing accuracy is improving every epoch or ...
user avatar
  • 287
1 vote
0 answers
20 views

Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
user avatar
  • 519
0 votes
0 answers
24 views

"Predictive dependence" between two variables

Given two random variables $X$ and $Y$, it is natural to use the conditional entropy $H[Y|X]$ to quantify the extent to which knowing $X$ decreases the uncertainty about $Y$. However, consider the ...
user avatar
1 vote
1 answer
21 views

Logistic regression predictions dont work

I have this problem with logit, that when I want to create confusion matrix, it simply displays the real values in the first row and in the second row, there are never any numbers. I created a lot of ...
user avatar
  • 11
0 votes
0 answers
9 views

How to increase accuracy and to receive better forecasts?

I am building a LSTM network and I am using the following structure: ...
user avatar
1 vote
1 answer
22 views

Accuracy failure example

I am trying to understand the functioning of accuracy and I need a practical example. This is what I understood: it gets the average correctness of the predictions and in some cases its result can be ...
user avatar
0 votes
0 answers
27 views

Accuracy for unfairness detection

I am dealing with unfairness and I am trying to find out some metrics to detect unfairness presence in my dataset. I am starting from the very bases: accuracy. My main question is Do you think I can ...
user avatar
0 votes
0 answers
17 views

What does it mean if the validation accuracy is equal to the testing accuracy?

I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
user avatar
  • 13
2 votes
1 answer
37 views

Will threshold of logistic regression change accuracy? Any relationship with the incidence of disease? [duplicate]

I am using a logistic regression model to predict breast cancer. I trained and tested the model in a population with a pretty high incidence of breast cancer(since the individuals all went to the ...
user avatar
1 vote
1 answer
18 views

Which best model I have to use to calculate standard deviations?

Suppose that I run the same experiment 5 times. Now, suppose that each model of each experiment has the best accuracy in different epochs. For example: Model1: best accuracy after 10 epochs Model2: ...
user avatar
1 vote
0 answers
62 views

Why did cross validation accuracy suddenly decrease? and then increase again?

I am training a classification model in Tensorflow. Here is a screenshot of the graphs I obtained after training. The blue one is the training accuracy, while the orange one is the cross validation ...
user avatar
0 votes
0 answers
6 views

Interpreting validation loss and accuracy for various learning rates

I am having a hard time comparing the effect of different learning rates on validation loss and accuracy. Would I be right in assuming that a Learning rate of 0.0001 was the most successful as the ...
user avatar
0 votes
1 answer
18 views

Understanding the idx column in h2o metrics [closed]

h2o model metrics results report generated like this. ...
user avatar
  • 3
0 votes
1 answer
36 views

How to evaluate te testing data from the trained data?

I am new at ML, and still trying to understand some concepts, so I figured I could ask here and maybe finally understand. How does it work the whole splitting data from a data set? Before answering, ...
user avatar
1 vote
1 answer
30 views

Why is a 3-copula better than a 2-copula?

Suppose that I have known that $X$ and $Y$ have high dependency, $Y$ and $Z$ have high dependency, and $Z$ and $X$ also have high dependency through three different 2-copulas. Suppose I fit one 3-...
user avatar
0 votes
0 answers
10 views

Metric to estimate measuring accuracy

I need help/suggestions on which metric/approach to use. I am trying to estimate pointing precision by measuring the offset of the experimentally measured values from the real, theoretical values, ...
user avatar
2 votes
2 answers
65 views

A/B test with a result of another deep learning analysis with 90% accuracy

I am planning to conduct a A/B test with data obtained through a deep learning algorithm. Say, I got a binary classification dataset through machine learning with about 100k rows classified into yes, ...
user avatar
  • 53
6 votes
2 answers
193 views

Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss

Vladimir Vapnik wrote: “When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one.” ...
user avatar
0 votes
1 answer
40 views

Compare model accuracy with null accuracy

My current model of Logistic Regression return a accuracy of: Training set score: 0.8476 Test set score: 0.8502 Comparing model accuracy with null accuracy: ...
user avatar
1 vote
0 answers
22 views

Which Proper Scoring rule for when? [duplicate]

Hi, I'm quite new to statistics and have been tasked to evaluate if there is a difference in accuracy between 2 subpopulations in a logistic model. The credit scoring company's model calculates the ...
user avatar
0 votes
0 answers
15 views

Assess results of non-linear continous model

I have a set of y_true labels and a set of y_pred labels. The prediction model is actually a deep learning model in combination with a rule-based information extractor. It tries to find the ...
user avatar
  • 1
0 votes
0 answers
29 views

Determine test accuracy

I'm trying to analyze results of a certain study. In this study, a certain diagnostic test was conducted in order to see if the pre-test assumed diagnosis was correct. Management was changed in about ...
user avatar
0 votes
0 answers
17 views

What metric can be considered to find if causal impact has determined the best combination of synthetic controls

I'm new to Causal impact. I read the paper and video by Kay which has a detailed description of the package. Can someone suggest any metric which can describe the accuracy of the synthetic control ...
user avatar
0 votes
1 answer
25 views

How do I obtain a percentage accuracy for an LSTM

I've got an LSTM trained for time series forecasting and I've seen people online report their LSTMs accuracy in %s such as 85% accurate etc, how do I obtain a metric like this? so far I was just using ...
user avatar
1 vote
0 answers
23 views

multi class prediction metric

I would like a method that generalized metrics for multiclass. Including imbalanced classes. This metric should be all against all. I thought of Generalizing the Mathew's coefficient but not in the ...
user avatar
0 votes
0 answers
73 views

Explain MAAPE (Mean Arctangent Absolute Percentage Error) in simple terms (intermittent demand forecasting)

In order to measure the accuracy of highly intermitted demand time series, I recently discovered a new accuracy measure, that overcomes the problem of zero values and values close to zero, when ...
user avatar
0 votes
0 answers
12 views

Regression vs Classification at Accuracy

Let's assume we want to predict some continuous values within y_regression in [-100,100], and all the X (train set) is continuous. Now, let's say we care only about ...
user avatar
1 vote
1 answer
29 views

Is Average Accuracy An Ill-Defined Metric? [closed]

Five people each took a 100 question yes/no question test. I have their individual accuracy scores and was planning to report their average. However, I was informed that "You cannot take an ...
user avatar
  • 33
0 votes
0 answers
8 views

How can I find the percentage accuracy between an approximation and a source formula?

Im trying to find the percentage of error for a trigonometric sine graph. I wrote an approximation of the graph, and am comparing the two up close, https://www.shadertoy.com/view/NslfRs Red represents ...
user avatar
0 votes
0 answers
23 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
user avatar
  • 3
1 vote
0 answers
34 views

the value of accuracy and loss change by the order of the training

https://colab.research.google.com/drive/13wMNCXxKs_uqFVzuJqE2zxhsxWK8Ya4k?usp=sharing Hello guys I’m having a hard time trying to figure out what I am doing wrong here. I used 4 pretrained models from ...
user avatar
1 vote
1 answer
82 views

Balanced accuracy reduces to accuracy for balanced datasets

This question might be trivial, but I have problems understanding this line taken from here: The balanced_accuracy_score function computes the balanced accuracy, which avoids inflated performance ...
user avatar
  • 113
1 vote
1 answer
118 views

calculating the top k accuracy using logits vs softmax probabilites

I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
user avatar
  • 11
1 vote
0 answers
15 views

What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]

As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting. Consider a case of ...
user avatar
0 votes
0 answers
23 views

Comparing classification metrics among studies

I am doing some comparison of classification results obtained by various studies using machine learning. All studies use the same "type" of data (satellite imagery) but differ in many ...
user avatar
  • 21
1 vote
2 answers
155 views

Reporting the best test accuracy in a research paper?

I am running a random forest classifier for binary classification. I have split the dataset into training and testing. From training data, I select 70% of the data for hyperparameter tuning, I used 5 ...
user avatar
  • 139
2 votes
0 answers
25 views

Training Set: Why Loss Flatten but Accuracy continues to increase?

I took a the Coursera course: Convolutional Neural Networks in TensorFlow, and one of the quiz questions is When exploring the graphs, the loss levelled out at about .75 after 2 epochs, but the ...
user avatar
  • 131
1 vote
0 answers
17 views

Comparing models with partially observed data and limited supervision

Setup: Suppose we have $n$ questions and for each question $m$ answers. For each question, a model, Model A, selects one answer out of $m$. Note that for each question, zero, one, or more answers ...
user avatar
  • 670
0 votes
1 answer
54 views

Accuracy, Recall (sensitivity), Specificity confusion

I came up with this paper with 765 citations! On page 2, it expresses an equation relating these metrics: Accuracy sensitivity (a.k.a Recall). specificity (a.k.a 1 - type1 error rate) prevalence (...
user avatar
  • 325
0 votes
1 answer
66 views

Does it matter if real data will be imbalanced, if the ML model was trained on a balanced dataset?

I have trained a machine learning model (supervised, classification, LinearSVC) on a balanced dataset, which produces relatively good results on the test data. I am happy with the numbers, but not ...
user avatar
1 vote
0 answers
31 views

Underfitting, overfitting or good fit?

I am working with a medical dataset, trying to figure out people with diabetes. I implemented LSTM since the target variable is a categorical one, the accuracy gave %84. I plotted the accuracy ...
user avatar
0 votes
0 answers
24 views

Is it appropriate to use difference scores in this context?

I am trying to compare the level of rating accuracy of two groups of participants (say Sample A and B). The study design is as is: I asked a consumer panel to rate how much they liked three products. ...
user avatar
  • 1
0 votes
0 answers
82 views

Can you calculate recall/precision/f1 score for "continuous variables"

My goal is to calculate some performance metrics (precision, recall, f1,accuracy) on a binary variable (yes or no) but have dates associated with each data point. Attaching dates to this binary ...
user avatar
  • 1
0 votes
0 answers
74 views

How to increase the accuracy of a multivariate LSTM network?

I am using a pollution dataset with values for pm2.5, pm10 and pm1 as features and I am predicting the values for the pm2.5. I built an LSTM network but the predicted values are quite from the real ...
user avatar
0 votes
0 answers
14 views

What social/economic/political domains are typ. easier-harder to forecast ? Are there benchmark estimates of accuracy for such time series forecasts?

For the report of the results from the Behavioral and Social Science Forecasting Collaborative Tournament I organized during the first COVID year, our team is trying to identify good benchmarks for ...
user avatar
1 vote
1 answer
57 views

ARIMA accuracy measures, rolling forecast

Regarding ARIMA model selection and especially accuracy measures several questions came into my mind. To shortly summarize, in my understanding, after necessary transformations/differencing, p and q ...
user avatar
  • 59
1 vote
0 answers
16 views

how to increase location accuracy?

I am developing a website for student attendance if a student is present in classroom in given time and he/she fill attendance by pressing a button then his/her attendance should be marked as present. ...
user avatar
  • 11

1
2 3 4 5
16