Accuracy of an estimator is the degree of closeness of the estimates to the true value. For a classifier, accuracy is the proportion of correct classifications. (This second usage is not good practice. See the tag wiki for a link to further information.)
1
vote
1answer
11 views
Evaluate classifier based on predicted probabilities?
If I had a categorical response $Y$ and multiple categorical features $X$, and I wanted to fit a model to predict $Y$.
If all I cared about was the eventual distribution of $Y$ (say in terms of %), I ...
1
vote
0answers
15 views
How to correctly compare the accuracy of different forecasting methods
I am currently working on a forecasting project and I have tried several different models to forecast with. Having trained and tuned my models I want to pick which model works best for each time ...
0
votes
1answer
15 views
Polluting a dataset with “non-determinables”
Let's say I'm working on solving sudoku puzzles with machine learning. Now, plenty of good methods exist for solving sudoku algorithmically, no machine learning required, but let's play along to get ...
0
votes
0answers
8 views
Can ICC be used to measure and benchmark ordinal classification accuracy?
I am conducting ordinal regression with an ordinal categorical target outcome roughly equally distributed as 1, 2 or 3. (This ordinal categorical target is actually the conversion of a numeric target ...
1
vote
0answers
24 views
Boosted decision trees: in which situations are “deep” decision trees performing better?
The general idea of boosted decision trees is to use very simple trees in the following manner (simplified, for intuition only):
start with a simple tree,
fit another simple tree on the residuals,
...
0
votes
1answer
38 views
Checking the similarity of two classes in a binary classification
So, I have a binary classification problem. The classes are fairly balanced and I have a separate training set and a test set. No matter what I try, both classification accuracy and the f1 score are ...
2
votes
1answer
60 views
How to meaningfully compute the accuracy of a multi-step forecast produced by a model
I am trying to measure the accuracy of my model in producing a multi-step forecast and I have read a lot of different opinions on the matter and am now rather confused.
The goal of my model is to ...
0
votes
0answers
25 views
Why is my ROC curve above the random line but the AUC is very low?
A little bit of background on my development process. I'm using a Random Forest model using the software package Alteryx (R Based) to classify a binary target variable that is approximately 60 (Neg - ...
0
votes
2answers
32 views
Can I create a test dataset with known errors to validate accuracy assessment? [closed]
I developed a procedure to measure the geometric accuracy of 3D building models based on the similarity to a 3D point cloud. Therefore I created mainly two quality criteria. The result of my automatic ...
0
votes
0answers
13 views
Compare accuracy between tools using k-fold cross validation, each tool is tested with different k values
I'm working on a new way to do the classification in a supervised way and I want to compare its accuracy to some related works. These works are using the same data set and they are testing their ...
1
vote
0answers
47 views
Calculation of accuracy (and Cohen's kappa) using sensitivity, specificity, positive and negative predictive values
I read How to calculate specificity from accuracy and sensitivity, but I have two diagnostic performance measures more. Please correct me if I am wrong: if
Sensitivity=TP/(TP+FN)
Specificity=TN/(TN+...
1
vote
1answer
19 views
Quality metric for classifier with decision rule allowing “none of the above”
Let's suppose that I have classification model for n classes ($n>1$). The classifier returns a probability distribution over a set of classes. But if classifier is not sure (i.e. there is no ...
0
votes
0answers
9 views
Fuzzy String Matching and Word Length
I'm attempting to calculate how likely it is that two words are the "same", but misspelled. I've narrowed it down to a few algorithms that will give me a similarity metric, but the problem is that ...
0
votes
0answers
35 views
Confusions about Pesaran & Timmermann test (2009 version)
I am learning about Pesaran Timmermann test (2009) for directional accuracy and I have troubles understanding its formula. Here I use notation and arrangement used in Pönkä (2017):
$$PT=(T-1)(S^{-1}_{...
0
votes
1answer
26 views
Unbalanced dataset accuracy [duplicate]
I’m currently encountering some problems analyzing a dataset with neural network. The problem is that I have an unbalanced binary class training set (10:1). Training accuracy for both classes are 100%....
0
votes
0answers
38 views
AUC and accuracy interpretation
An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as:
Given a sample whose outcomes we want to predict, 80% of the prediction will be correct.
What does an AUC of $80\%$...
0
votes
0answers
10 views
Importance of standard deviation in a classifier
I have two different classifiers that affect the same dataset. I ran a repeated 10-fold cross validation and these are the results:
C1:
total mean of precision = ...
2
votes
1answer
41 views
Constant Accuracy with decreasing loss
I am fairly new to Cross Validated section so I apologize if my question structure is incorrect.
I am currently working on Fully Convolutional Networks for Semantic Segmentation.
I am first trying ...
0
votes
1answer
34 views
reconstruct a 2X2 confusion matrix (TP, TN, FP, FN) from Sensitivity and Specificity
Is it possible to reconstruct a 2X2 confusion matrix (TP, TN, FP, FN) from Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Values. I also have prevalence according to the ...
2
votes
1answer
60 views
Classification model accuracy, roc auc score, f1 score 100%
I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't ...
2
votes
1answer
103 views
Example when using accuracy as an outcome measure will lead to a wrong conclusion
I am looking into various performance measures for predictive models. A lot was written about problems of using accuracy, instead of something more continuous to evaluate model performance. Frank ...
1
vote
0answers
30 views
How to make really bad results from a machine learning model better by reversing predictions
I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
0
votes
3answers
47 views
accuracy of a binary classifier vs probability of binomial distribution
I got confused over a simple concept.
Imagine that I have a binary classifier with 50% accuracy. So, if there are 10 samples to be classified as "y", "n", it has predicted 5 of them correctly.
Now, ...
0
votes
0answers
16 views
Evaluating accuracy of the classifier based on sample?
I have a rather odd question which unfortunately goes beyond my knowledge of stats so any advice will be much appreciated. We built a clustering model on the text data (LDA) and then assigned classes ...
1
vote
1answer
204 views
Difference between Mean/average accuracy and Overall accuracy
I just got confusion while reading the paper "Local Binary Pattern-Based Hyperspectral Image Classification With Superpixel Guidance".
They mentioned that they repeated each experiment 10 times and ...
1
vote
1answer
31 views
Does high training accuracy for an NN mean that it has a potential to reach high validation accuracy?
I saw quite a few discussions related to the problem of high training accuracy with low validation accuracy and what steps to take to address it. I have the same problem with a binary classification ...
3
votes
1answer
39 views
Quantitative evaluations for image classification
Hello I am working on the classification of different weed categories. I want to know what quantitative evaluation I can do other than find the accuracy?
0
votes
1answer
32 views
why does data external from training and testing my neural network perform much worse than statistical accuracy?
I've got a problem with my neural network (used to recognize audio signals, an expansion of the UrbanSound dataset problem): when I fit the model the accuracy of both train and validation is near 90%. ...
2
votes
0answers
33 views
Can F1-Score be higher than accuracy?
I'm using sklearn's confusion_matrix and classification_report methods to compute the ...
1
vote
0answers
57 views
Accuracy metric for comparing Time Series models?
I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below).
All ...
0
votes
0answers
7 views
Can the size of the test set affect the results?
I have a language model, a bi-LSTM, which predicts the next word.
I divide the corpus in 80% training, 10% development and 10% test and I obtain 51% accuracy. Afterwards I wanted to test with a ...
0
votes
1answer
46 views
Incomprehensible lowering of VGG19's accuracy if data is preprocessed
I'm using the VGG19 convolutional network for image recognition (to be specific, the keras implementation). I've downloaded the ILSVRC2012 validation dataset and the ground truth from here, to check ...
0
votes
0answers
6 views
What features or architecture for a text answering system?
From this dataset with paragraphs, questions about these paragraphs and answers from these paragraphs, when there exists, I'm trying to predict the sentence where there is an answer. After ...
1
vote
1answer
149 views
What influences fluctuations in validation accuracy?
This is the Tensorboard output of some machine learning experiments I'm doing.
What are the factors that influence the entity of the fluctuations in the validation accuracy? For example: the batch ...
2
votes
1answer
143 views
Improving spam classification with tensorflow logistic regression
I would like to classify a mail (spam = 1/ham = 0), using logistic regression. My implementation is similar to this implementation and using tensorflow.
A mail is represented as a bag-of-words ...
0
votes
1answer
29 views
Accuracy percent seems too high in Neural Network
I built an artificial neural network that has a dependent variable called "Suspicious". This column is binary so only two outcomes. I have 297,771 "0" not suspicious or known good. Then I have only ...
0
votes
0answers
26 views
reconstruction loss for disentangled variational autoencoders
I am using disentangled variational autoencoders which is a variant of VAE. You will find the github code in this link.
I want to quantify the difference or the loss between the ground truth (...
-1
votes
1answer
81 views
0
votes
1answer
39 views
Is there an approach to predict the accuracy of a model without the test_labels? [closed]
I am working with the Loan Prediction Problem. My testset however does not contain the test labels. Is there any way in which accuracy score could be obtained without the test labels?
0
votes
0answers
24 views
When do we have equal accuracy and f1-score in binary classification?
I am working on a link prediction method and when I run the code, the output of accuracy and f1-score are equal in all iterations. I can't interpret the results. The number of positive and negative ...
-1
votes
0answers
8 views
Feature selection using PArticle swarm optimization
I want to perform classification using weka tool but I am keen to do feature selection using PSO. I am using cocomo dataset for ...
0
votes
0answers
45 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
6
votes
1answer
201 views
Is accuracy an improper scoring rule in a binary classification setting?
I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
0
votes
0answers
37 views
Keras accuracy not affected by any change in model [duplicate]
I am trying to construct a model for single-label multiclass classification using Keras in a Jupyter notebook. Here's my model (or see full jupyter notebook):
...
2
votes
1answer
108 views
Calculating the accuracy of matching between two sets of strings lists
To give some background - the question is about measuring the accuracy of name disambiguation algo results (and not about the algo itself)
Let's say we have three groups of entities, each ...
2
votes
2answers
17 views
Distribution of weighted accuracies for classification
A long time ago I seem to remember learning that rather than just assigning a 0 or 1 accuracy values for samples, one could calculate the weighted accuracy of a sample based on the confidence level of ...
0
votes
1answer
41 views
Educational purpose :Calculation of sensitivity and specificity from confusion matrix for this example
Consider a case where the number of labelled data as 0 = 1400 and labelled as 1 =100. The data labelled as 0 denote normal operating conditions and data labelled as ...
-1
votes
1answer
66 views
Clustering evaluation metric when overclustering is of [closed]
What evaluation metric should be used to measure clustering performance when over-clustering is OK as long as it happens only within ground truth clusters, with no confusion of ground truth clusters.
...
1
vote
0answers
14 views
Accuracy in multi-label classification [duplicate]
In a multi-label classification, the accuracy is commonly defined as [1]
$$
\text{Accuracy}(\boldsymbol{Y},\, \boldsymbol{Z}) := \frac{1}{n} \sum_{i=1}^n \frac{\lvert Y_i \cap Z_i \rvert}{\lvert Y_i \...
0
votes
0answers
19 views
Is there a way to find the overall probability of the model fitting if it fits all models?
I have developed 4 models, using machine learning.
I have been working on using machine learning on molecular fingerprints, to determine if new compounds are active or inactive.
If i have 4 models, ...
