Questions tagged [confusion-matrix]
A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.
290
questions
0
votes
0
answers
18
views
What should I do when a test specificity is zero?
Upon doing a test that compares the effect of imaging versus regular tissue biopsy for cancer that has a prevalence of 90%, we ...
- 211
10
votes
3
answers
808
views
For a confusion matrix, is there a name for FP / (FP + FN)?
For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about:
FP / (FP + FN)
Of course this measure is ...
- 203
1
vote
1
answer
14
views
Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?
Given the following table of predictions vs. actual states:
...
- 1,147
0
votes
0
answers
17
views
Confusion matrix : Values for TN are negative
I am trying to get the values for True Positives, False negatives, false positives, and true negatives for my confusion matrix when I run the k-nearest algorithm in Matlab. However, when I calculate ...
0
votes
0
answers
65
views
Overfitting models in mlr3
I'm trying to compare multiple learners on my dataset (called "data") in order to predict a target called "lesionResponse", with custom resampling. Since mlr3 package doesn't allow ...
- 3
1
vote
1
answer
50
views
Customized F1-Score for multi-class classification
Let's consider a multi-class classification problem with 4 classes: 0, 1, 2, and 3
F1-Score 'macro'-averaged is calculated like that:
...
- 47
3
votes
0
answers
62
views
Confidence Informed Confusion Matrix (Threshold Free)
I am using the normalized confusion matrix to aid in quantifying the uncertainty in related observations over time. More specifically, I have a classifier that returns the confidence of each class via ...
- 31
1
vote
1
answer
43
views
Surprising disparity between Confusion matrix values and AUC?
I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
2
votes
0
answers
53
views
Can Accuracy be higher than both sensitivity and specificity?
I came across a paper which reported the following results
Accuracy
Specificity
Sensitivity
97.49%
93.6%
94.3%
It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
- 131
11
votes
8
answers
6k
views
My machine learning model has precision of 30%. Can this model be useful?
I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
- 121
0
votes
0
answers
35
views
Logistic regression model with different predictors giving same Confusion Matrix
I am new to statistics and regression model and still trying to grasp the concepts of models and its accuracy.
One of my assignments is to create a logistic regression model in R using a tele company'...
0
votes
1
answer
42
views
Confusion Matrix with a class created from lower probabilities of other classes
Please comment on the following method:
Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here.
Then ...
- 277
5
votes
2
answers
274
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
- 115k
1
vote
0
answers
47
views
How to measure sensitivity, speficity, PPV and NPV from confusion matrix for multiclass classification [duplicate]
I want to calculate sensitivity, specificity, PPV, and NPV from a confusion matrix for multiclass classification. My objective is to learn the basic concept of constructing a confusion matrix for ...
- 11
0
votes
0
answers
36
views
Levels error in complexity matrix
I am trying to classify in R with neural network and Gradient Boosting. My target variable is normally low, medium and high. I'm trying to classify using the codes below, but when I want to create a ...
- 13
0
votes
0
answers
28
views
How can you adjust the values in a pair-confusion matrix for chance?
In cluster analysis one of the entries of a pair-confusion matrix is the number of data point pairs that are grouped together by both clustering algorithms.
How could one adjust this value for chance ...
1
vote
3
answers
84
views
Proof that Recall ≥ F1-score
I need to prove mathematically that the value of Recall is always bigger or equal to the F1-score.
The equation would be: $2 \times Precision \times \frac{Recall}{(Precision + Recall)} \le Recall$
...
- 13
1
vote
1
answer
134
views
How is it possible to get AUC = 0.72, but sensitivity = 0?
I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good:
However, using the same test data, this is the result of a ...
- 433
0
votes
0
answers
43
views
Cross-validation and F1 metric
Cross-validation with metrics such as F1 can be implemented in two ways:
For each cross-validation split, calculate F1_split on the validation dataset.
F1_result = average_by_splits(F1_split)
For ...
1
vote
1
answer
46
views
Metric to evaluate binary classification model on imbalanced dataset in order to meet percentage limitations
For a university class I'm working on a imbalanced dataset that has ratio of 43:1 Class_0 to Class_1.
Class_1 refer to companies that have declared bankruptcy based on feature columns of the dataset. ...
- 11
0
votes
0
answers
20
views
Performance assessment based on conditional probability with a reference model
Say I have a reference binary classifier model with its' confusion matrix. Then I construct another model and I can assess the probability they agree (relative confusion matrix). Is there any way to ...
- 360
0
votes
0
answers
11
views
Is there a way to determine the size of a condition positive population?
I am currently working on a project where we are using a random forest classifier (SciKit-Learn's specifically) and we are trying to verify that all three versions we have of the classifier are ...
- 1
2
votes
2
answers
232
views
Sensitivity vs. specificity vs. recall
Given a binary confusion matrix with true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), what are the formulas for sensitivity, specificity, and recall?
I'm ...
- 219
1
vote
2
answers
187
views
Is there a version of accuracy weighted by prevalence?
I have been learning the basic terminology for how to think about binary tests involving medical tests. The basic terms are here in this table
This is the confusion matrix.
My issue is the following. ...
- 3,913
5
votes
2
answers
504
views
Can I apply a confusion matrix to classification tasks outside of ML?
I would like to know if it's possible to use a confusion matrix to measure the performance of a classification tool outside the realm of ML or a statistical model.
For example, if I had a small script ...
- 51
0
votes
0
answers
46
views
Harmonic mean of false positive and false discovery rates (analogous to F1)
F1 is the harmonic mean of recall (aka sensitivity, or true positive rate, TPR) and precision (aka positive predictive value, PPV).
$\text{TPR} = \text{Pr(predicted:Pos | Pos)} =$ TP/P (wikipedia ...
- 101
5
votes
2
answers
412
views
Poorly calibrated probabilities but good classification in confusion matrix
I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/...
- 265
1
vote
0
answers
28
views
Formula versus Non-Formula Interface Categorical Variables train() glmnet
I am comparing the confusion matrix between the formula interface and the non-formula interface using caret's train() for elastic net. I am trying to understand why the two interfaces produces ...
- 265
3
votes
1
answer
396
views
Why is "balanced accuracy" an arithmetic mean instead of harmonic?
In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
- 11.8k
0
votes
1
answer
488
views
How to analyse results of classification for time series + sliding windows
Here is my context:
I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows.
...
- 9
0
votes
1
answer
411
views
How to calculate the accuracy and error rate in confusion matrix?
(0 - negative outcome, 1 - positive outcome)
Am I correct? Because I found a different answer.
My answer:
Accuracy rate: TP+TN/ALL(58+25/58+44+23+25)=83/150
Error rate: FN+FP(23+44)= 67/150
Someone ...
- 21
0
votes
0
answers
33
views
When I convert the result of 'predict' function of logistic regression into factor, should the cutoff be same with the that of dependent variable? [duplicate]
I wanna run glm composed of dependent variable named "exposure", and independent variable named "counts" and "distance" respectively.
...
2
votes
1
answer
40
views
A metric for a big/medium/small ML classification
I am working on an ML classification task which is similar to the following:
Apples have to be classified to three classes: Big, Medium and Small.
I need a metric which I can use to assess the system. ...
- 123
0
votes
0
answers
35
views
When are we satisfied with the performance of a classifier?
I am training a multi-head classifier (based on Resnet18) for a multilabel classification task. The dataset I am using to train the classifier is noisy and to improve the accuracy for the worse-...
- 137
1
vote
0
answers
231
views
Are Type I and Type II errors in multiclass problem appear to be the same?
As far as I know from binary classification
FP error is a type 1 error
FN error is a type 2 error
I have this confusion matrix generated:
And here I found how to read this confusion matrix:
As you ...
- 111
1
vote
0
answers
78
views
How to assign costs to the confusion matrix
I am trying to assign costs to the confusion matrix. That is, in my problem, a false positive (FP) does not have the same cost as a false negative (FN), so I want to assign to these cases a cost "...
- 135
3
votes
1
answer
97
views
Two confusion matrices
In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to $...
- 1,649
0
votes
0
answers
36
views
Understand AUCs of different models
I'm testing two models against each other:
provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %.
provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
- 3,123
0
votes
0
answers
25
views
Make ROC graph with my problematic [duplicate]
For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic.
I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
- 1
1
vote
1
answer
48
views
How to interpret this confusion matrix and roc curve?
I got these two images for evaluating a RF:
I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %?
By looking only at ...
- 3,123
1
vote
1
answer
752
views
Logistic Regression: What is the value for precision when recall (true positive rate) is 0?
A quick overview of definitions before I get into the question:
True Positive (TP): An actual positive that the model classified as positive
False Positive (FP): An actual negative that the model ...
- 641
1
vote
0
answers
266
views
Confusion matrix error [closed]
I have been trying to do support vector machine classification in R and confusion matrix has been showing the error even after changing into factors
confusionMatrix(as.factor(yTestPred), as.factor(...
- 11
3
votes
2
answers
2k
views
why I am getting TP and FP in confusion matrix is 0, how to get it right?
why I am getting TP and FP in confusion matrix is 0, how to get it right?
tp = 0,fp = 0,tn = 9847,fn = 18
1
vote
1
answer
34
views
Estimate sample size from a variable population
Context:
I want to measure accuracy, precision, and recall for individual raters. Each rater completes a variable amount of labels, for ex. rater A may complete 500 in a given time period while rater ...
1
vote
1
answer
719
views
Creating confusion matrix and calculating TP, TN,FP, FN
I want to generate a confusion matrix for class 2. There is a total of four input classes namely 0,1,2,3. In particular test data batch, there is no input ...
- 147
1
vote
3
answers
4k
views
My training accuracy is 1.0 and my test accuracy is 0.994. Am I overfitting for a multiclass classification?
This is a multiclass classification for an imbalanced dataset. I set the class_weight for this model to "balanced". I have a perfect training accuracy (1.0) and a nearly perfect testing ...
- 11
1
vote
1
answer
217
views
How to compare ctree and cforest approaches on the classification task?
I doing some numerical experiments with ctree() and cforest() functions from the partkid package.
I am using the Wine Quality Data Set.
...
- 848
0
votes
1
answer
105
views
In confusion matrix prediction, has the training data used for prediction been seen by the model
I understand that Confusion Matrix evaluates prediction results with metrics such as precision, recall, accuracy and F1-Score.
My question is regarding the data used for prediction. Is it okay if this ...
- 107
3
votes
1
answer
112
views
When comparing classifiers on different datasets with different prevalences, is it valid to calculate prevalence-adjusted PPV?
Scenario: comparison of 2 different binary classifiers
Both classifiers report sensitivity and specificity and number actually positive (P), but classifier 1 is tested on a dataset with prevalence 20%,...
- 31
1
vote
1
answer
505
views
Why does confusion matrix shows inverse output?
I am working on a logistic regression binary classification with 1000 rows and 28 columns
While I have split my dataset into train and ...
- 2,894