Questions tagged [accuracy]

Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Error in accuracy test [duplicate]

This is an update from my previous question. I'll put my Model Development code here for your reference: ...
Amsyar Nifail's user avatar
0 votes
0 answers
7 views

Binary Classification - Separating data by categorical data and creating regression for each group

My and friend and I are complete beginners to statistical learning and we were playing around with a data set on loan approval using logistic regression. Each data point contains numeric variables ...
victor su's user avatar
1 vote
1 answer
54 views

Distribution of accuracy from randomly guessing

Let's consider a true classification problem, that is, one where the predictor makes categorical predictions (not probabilities). It makes sense to assess the accuracy of such a predictor. However, ...
Dave's user avatar
  • 55.1k
6 votes
3 answers
2k views

Is it okay to say that 95% confidence interval is more significant than 80%?

So, the higher the confidence interval the lower the false positive rate, but the false negative rate will increase lowering the recall. Is it possible to determine which confidence interval is better/...
Ankita's user avatar
  • 119
2 votes
0 answers
84 views

When *is* classification accuracy the right measure of performance

Plenty has been discussed on Cross Validated about the drawbacks of classification accuracy when it comes to evaluating classification models. One good answer is here, for instance. Under what ...
Dave's user avatar
  • 55.1k
1 vote
0 answers
13 views

Improve accuracy randomForest classification model [duplicate]

How do I improve the accuracy of the following data. It is from the following Kaggle competition which I am doing (despite it being closed for a school project). ...
Imme's user avatar
  • 11
1 vote
0 answers
17 views

PLSR: trait vs spectroscopic data gives very low R2 on plsr model in R

here is the sample data. I have spectroscopy data as X-variables (from X1 to X80) and corresponding Y variable. I need to run plsr model in R using "pls" package. There are two sheets. In ...
MGD's user avatar
  • 11
0 votes
0 answers
14 views

Did I lose precision when I applied inverse normalization and inverse standarization when I predict the new data in a regression problem?

Let's suppose that I'm trying to predict a stochastic forecast with machine learning models, and I don't have missing, null/NaN values and outliers. Also suppose that there is an error for the ...
Daniel_DS's user avatar
1 vote
2 answers
31 views

why does a model with a larger val loss produce higher accuracy than a model with a smaller val loss?

I did ANN classification on training data with oversampling and without oversampling. For each data, the smallest validation loss is sought with trial and error of 18 models. In the data without ...
andryan86's user avatar
  • 117
0 votes
1 answer
35 views

Looking for a formula to determine accuracy of a sample

I am looking for a formula to help me determine the accuracy of a population. Here is my business problem. I have about 1 million scanned documents of many types that are currently unclassified ...
Alex 's user avatar
0 votes
0 answers
14 views

p-value for results of sub group analysis with ML?

I developed a ML algorithm (Xgboost) to predict a target in my data set. I obtain here the results of my predictions on my test set : ...
Nicolas's user avatar
0 votes
0 answers
24 views

Accuracy on NN model decrease after random oversampling using library ROSE

I did random oversampling to handle unbalanced positive and negative data. When I didn't do random oversampling, the accuracy I got was 88%, when I oversampled the train data, it got 87% accuracy and ...
andryan86's user avatar
  • 117
0 votes
1 answer
80 views

How does someone achieve a desired confidence / accuracy when measuring using uncalibrated instrument?

I have an instrument that measures a value. It is only possible to measure the value once i.e. the experiment can't be repeated (think recording a car's speed as it drives past). The instrument is not ...
Chuck's user avatar
  • 81
1 vote
0 answers
65 views

Confidence intervals for binary classification

I'm doing binary classification in Python with an SVM classifier, and I implemented stratified repeated cross validation to have more robust results. I would like to calculate confidence intervals for ...
Ed9012's user avatar
  • 41
1 vote
1 answer
45 views

Upper bound on classification performance

Given a set of 128x128 images from three classes, I obtained an accuracy of 50% with a SVM on the flattened images (16384 'features'). Is this an upper bound on the performance of a SVM using any ...
Christian's user avatar
  • 111
1 vote
0 answers
15 views

Different Estimates but same Out-of-sample predictive accuracy

I have two different models that give me different estimates on my data. The difference is not huge, but one model significantly shrinks the estimates towards zero. However, when I run leave-one-out ...
Zlo's user avatar
  • 137
2 votes
0 answers
36 views

Exactly when can I model a two-dimensional region with a 100% accuracy decision tree?

This is a theoritical question about decision trees. I believe that the question is very explicit in the title of the question. My thoughts on this question follow below. If the region we are dealing ...
xyz's user avatar
  • 155
0 votes
0 answers
26 views

Bayesian statistics: Inferring a true value for test sensitivity and specificity

Based on a data set I found a sensitivity (.82) and specificity (.88) for a diagnostic test bases on a n=257 sample. However, I wonder whether I can generalize these numbers. I thought this was very ...
Weflex's user avatar
  • 1
1 vote
1 answer
735 views

A binary classifier with accuracy below 50% and similar situations

Let's say, I have a binary classifier. Usually, if a model doesn't learn anything useful, the accuracy would be around 50%. Anything above 50 is better than a random guess. My question is: what are ...
Zabir Al Nazi's user avatar
2 votes
0 answers
53 views

Can Accuracy be higher than both sensitivity and specificity?

I came across a paper which reported the following results Accuracy Specificity Sensitivity 97.49% 93.6% 94.3% It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
Jack O'Neill's user avatar
5 votes
1 answer
378 views

Is Brier Score appropriate when comparing different classification models?

TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should ...
Luiz Gustavo's user avatar
0 votes
1 answer
45 views

High Cross Validation but low test accuracy on LibSVM

I am solving the problem of detecting swallowing and non-swallowing events from the audio. I labelled the data using Praat software by marking the swallowing and nonswallowing events. I trained the ...
Yalçın Cenik's user avatar
0 votes
0 answers
19 views

Why is the accuracy of a Decision Tree decreasing whereas the accuracy of an LSTM is increasing when adding augmented data?

I am using sklearn's DecisionTreeClassifier and LSTMs (Keras) for time series classification. To increase the accuracy and robustness of the models I augmented the ...
Unistack's user avatar
1 vote
0 answers
60 views

AUC ROC and accuracy for different datasets of same problem

I have datasets that correspond to different traffic load inputs. I am doing binary classification on them. The proportion of 1s to 0s varies from dataset to dataset. E.g., dataset 1 is imbalanced ...
knowledge_seeker's user avatar
1 vote
1 answer
69 views

How to know if features chosen are right?

How can we ensure that the chosen features can lead us to high accuracy if we made proper modifications in model architecture & hyper-parameters using the selected features, i.e. how can we make ...
Thunder's user avatar
  • 117
0 votes
0 answers
12 views

sensitivity-specificity and other test

I have two different test done in two different facilities. I want to measure the accuracy of each and compare. I have measured the sensibility, specificity, PPV and NNP of each test using a 2x2 table....
merco12's user avatar
4 votes
2 answers
169 views

Is the proportion classified correctly a reasonable analogue of $R^2$ for a classification model?

Let's do some classification and evaluate the prediction quality. The easiest metric to understand is the prediction accuracy, which can be reported as the proportion classified correctly to put the ...
Dave's user avatar
  • 55.1k
0 votes
0 answers
152 views

LSTM accuracy not increasing more than 68%

I am a newbie in the field of deep learning. I am trying to use LSTM with time series data with more than 300,000 datapoints. The target variable is a class with 7 unique values. I used one-hot ...
Hania's user avatar
  • 13
0 votes
0 answers
113 views

Confidence interval for Accuracy, Precision and Recall

Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that ...
dokondr's user avatar
  • 277
5 votes
2 answers
274 views

Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
Stephan Kolassa's user avatar
16 votes
2 answers
1k views

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Accuracy, as a KPI for assessing binary classification models, has major drawbacks: Why is accuracy not the best measure for assessing classification models?. The exact same issues also plague the F1 ...
Stephan Kolassa's user avatar
1 vote
0 answers
24 views

Much better results when standardizing features to train LSTMs

I have a data set of time series. Each time series represents trajectories of the same path taken. So, the time series captures acceleration in $x$, $y$ and $z$ direction, respectively for the ...
Invader's user avatar
  • 11
1 vote
1 answer
28 views

Having to use features which have low correlation with the target

I'm applying LogisticRegression on breastcancer dataset. Steps : - 1- A correlation matrix resulted in only four features having ...
Pixel_Bear's user avatar
0 votes
0 answers
12 views

Training accuracy stuck at .8 for convolutional neural network. How can I reduce the overfitting of my model? [duplicate]

I am creating a final project for a machine learning course. My project is a deep learning model that can classify images as either containing a weapon or not. Specifically, a gun vs other common ...
user1059114207's user avatar
0 votes
0 answers
45 views

Automatically obtain optimum cutoff point for accuracy, sensitivity and specificity

Below is a code snippet to plot the accuracy, sensitivity and specificity. I can manually eyeball and see the cutoff point at approximately around 0.3. I was wondering if there anyway to print out ...
Larry Chuon's user avatar
1 vote
1 answer
125 views

How to interpret R-squared versus nRMSE for a random forest model?

I have trained six random forest regression models (to predict topsoil, subsoil and total soil organic carbon stocks for two study ares) using out-of-bag validation, and I have gathered the R² and ...
a_big_chicken's user avatar
1 vote
1 answer
22 views

Accurracy for predicted vectors

I am currently working on a machine learning model that yields a vector of offloading decisions. An example: [-1, 0, -1, 1, 1, 0, ...] The model does not return this vector directly. Instead, the ...
YuKa's user avatar
  • 13
1 vote
1 answer
242 views

Accuracy without true negatives

Are there any named metric that calculates TP/(TP+FN+FP)? It is like the accuracy, but disregarding TN In my problem, we have thousand of possible labels. However, the models return a small list of ...
Jonathan alis's user avatar
0 votes
1 answer
82 views

What degree of difference does validation and training loss need to have to be called good fit?

I am conducting a multi-variate time series forecasting using an LSTM model. The model architecture and other details are given below: Dataset split: (80/10/10 split) Training Data Points: 367640 ...
Abdullah's user avatar
2 votes
2 answers
762 views

How many datapoints are enough for a regression model to predict with reasoanble (say 88%-92%) accuracy? [closed]

Is there any number that we can land on for our regression model to predict with high accuracy? (accuracy metrics I have in mind at RMSE or R-squared). Also high accuracy may mean something above 88% ...
SJa's user avatar
  • 524
1 vote
0 answers
16 views

Which one to choose: subject-wise or record-wise

I want to do some classifications on open access "Activity Recognition Using Wearable Physiological Measurements" data set. In this dataset, there are 40 subjects whose Electrocardiogram (...
Mina's user avatar
  • 111
1 vote
2 answers
800 views

Precision vs. specificity

I know that if we cannot afford to have false positive results, we should aim for high precision. My question is, how is precision different from specificity? Any examples?
MxML's user avatar
  • 21
1 vote
1 answer
73 views

How to evaluate luck vs skill in judgment accuracy and how to compare different measures of accuracy?

I have data about performance on two types of judgment task (for people), each type with a different format of ground truth for the targets (also people). All judges evaluated all targets, there were ...
guppy's user avatar
  • 23
1 vote
1 answer
816 views

Does weighted MAPE (wMAPE) provide an accurate estimate of error?

I'm trying to use wMAPE to calculate demand standard deviation for future forecasts, however, the wMAPE that we are using is derived from 13 buckets of actuals over a period of 52 buckets of time. If ...
NGSK's user avatar
  • 11
0 votes
0 answers
272 views

Multilabel Classification: Accuracy is very low. Metric or Model, which is inadequate?

In my multilabel classifaction problem, which I approach similarly to what can be see in this post: How does Keras handle multilabel classification?, the resulting accuracy only increases from 2% to 5%...
Viktor's user avatar
  • 101
1 vote
0 answers
257 views

What is the recommended goodness of fit test for binary logistic regression with large sample size?

I have a binary logistic regression with one unique independent variable X and several control variables. My data has over 240k observations. When I ran both HL and Pearson chi square goodness of fit ...
Laiy's user avatar
  • 205
1 vote
0 answers
24 views

why does SVM outperforms KNN in 1-gram but in 2,3,4, and 5 KNN outperforms SVM?

my project is authorship attribution which is a multiclass classification, the number of classes is 150, and the number of documents is 2798; it is also an unbalanced issue some classes have more ...
saya hassan's user avatar
1 vote
2 answers
285 views

Combine accuracy, precision, and recall

I am working on a classification problem. Several models are produced and all have accuracy, precision and recall metrics on test data. I need to pick the best model among the alternatives. What I can ...
Emin Ozkan's user avatar
1 vote
2 answers
205 views

How to split data as train and test set in a fixed manner?

I've been struggling to figure out the best technique to assess model accuracy in relation to the train/test split. Leave-one-out cross-validation and KFold appear to be more appropriate to utilize; ...
vdu16's user avatar
  • 13
2 votes
1 answer
211 views

Does it make sense to increment by 1 the numerator and denominator in the MAPE to avoid division by 0?

One of the drawbacks of MAPE is that you can have actuals equal to 0. This will result in a division by 0 and thus an undefined MAPE. Does it make sense to increment by 1 the numerator and denominator ...
lalaland's user avatar
  • 227

1
2 3 4 5
17