Questions tagged [confusion-matrix]
A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.
296 questions
0
votes
0
answers
18
views
Best way to solve a multi-class thresholding problem?
I have a multi-class classification problem, where I classify my images into n classes and also a background class. I am currently trying to figure out how to find the optimal thresholds (for each ...
- 1
0
votes
3
answers
137
views
False Negative vs False Positive for Multiclass classification
Suppose I have three classes 1,2,3.
And there's evaluation like below, where second element is false prediction where model predict class 3 while ground truth is 2.
...
5
votes
3
answers
563
views
Judging a model through the TP, TN, FP, and FN values
I am evaluating a model that predicts the existence or not existence of a "characteristic" (for example, "there is a dog in this image") using several datasets. The system outputs ...
- 257
0
votes
0
answers
140
views
Using Predictive Value Confidence Intervals to "Predict" Outcomes
Here's the quick version:
Say I have a confusion matrix with the following data based on a proficiency cut score on a pretest and outcomes on (passing/failing) a class. The cut score was determined ...
- 1
0
votes
1
answer
99
views
How to interpret this confusion matrix in R?
I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%.
These are the results of my confusion matrix:
Confusion Matrix and Statistics
Confusion Matrix (...
- 51
0
votes
0
answers
9
views
Is the confusion matrix meaning the same within deep learning than within hypothesis testing?
Is the confusion matrix meaning the same within deep learning than within hypothesis testing?
I've read what the matrix is, and how to get the values. But in the case of deep learning for ...
- 173
2
votes
3
answers
222
views
Is relying on just the confusion matrix for highly imbalanced test sets to evaluate model performance a bad idea?
I have a binary classification model with a test set that is highly skewed, the majority class 0 is 22 times greater than the minority class 1.
This causes my Precision to be low and Recall to be high,...
- 21
1
vote
0
answers
41
views
Help interpreting multi-class confusion matrix
I'm looking at the SAMHSA Mental Health Client-Level dataset. I did some t-SNE plots (dropping irrelevant cols, normalizing some, one-hot encoding some) of 500k rows out of 6.5mil.
I'm trying to do ...
- 163
4
votes
2
answers
153
views
Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)
This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
- 67k
3
votes
1
answer
47
views
Precision and recall in the case of null predictive power
With a multiclass classification problem, if it is perfectly predictive, the probability, $P(j|i)$, of predicting class $j$ for the real class $i$ is 1 when $j=1$ and 0 otherwhise. In this case the ...
- 141
0
votes
1
answer
103
views
Matthews' Correlation Coefficient and Leave-One-Out Cross Validation
How do I determine Matthews' Correlation Coefficient for leave-one-out cross-validation? There doesn't seem to be a way to average the coefficient across folds - one element is insufficient support ...
- 13
0
votes
0
answers
57
views
Drawing a dendrogram based on a confusion matrix
I have a simple 2D confusion matrix that I normalised for every true class. Now, what would be the best method to visualise this as a dendrogram?
I have seen hierarchical agglomerative clustering on ...
5
votes
1
answer
283
views
Is ROC curve unique?
ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
- 4,487
2
votes
1
answer
1k
views
Imbalanced classes and possible ways to increase precision, recall and f1-score of the prediction model
I've just started my data science internship, and this is my first time in the field. I'm sure I'll face challenges in the future where I might need your help. It's also my first time asking a ...
10
votes
3
answers
963
views
For a confusion matrix, is there a name for FP / (FP + FN)?
For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about:
FP / (FP + FN)
Of course this measure is ...
- 203
1
vote
1
answer
23
views
Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?
Given the following table of predictions vs. actual states:
...
- 1,291
0
votes
0
answers
116
views
Overfitting models in mlr3
I'm trying to compare multiple learners on my dataset (called "data") in order to predict a target called "lesionResponse", with custom resampling. Since mlr3 package doesn't allow ...
- 13
2
votes
1
answer
391
views
Customized F1-Score for multi-class classification
Let's consider a multi-class classification problem with 4 classes: 0, 1, 2, and 3
F1-Score 'macro'-averaged is calculated like that:
...
- 127
3
votes
0
answers
140
views
Confidence Informed Confusion Matrix (Threshold Free)
I am using the normalized confusion matrix to aid in quantifying the uncertainty in related observations over time. More specifically, I have a classifier that returns the confidence of each class via ...
- 31
1
vote
1
answer
141
views
Surprising disparity between Confusion matrix values and AUC?
I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
5
votes
1
answer
330
views
Can Accuracy be higher than both sensitivity and specificity?
I came across a paper which reported the following results
Accuracy
Specificity
Sensitivity
97.49%
93.6%
94.3%
It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
- 163
11
votes
8
answers
9k
views
My machine learning model has precision of 30%. Can this model be useful?
I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
- 121
0
votes
1
answer
62
views
Confusion Matrix with a class created from lower probabilities of other classes
Please comment on the following method:
Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here.
Then ...
- 287
8
votes
2
answers
475
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
- 131k
1
vote
0
answers
55
views
How to measure sensitivity, speficity, PPV and NPV from confusion matrix for multiclass classification [duplicate]
I want to calculate sensitivity, specificity, PPV, and NPV from a confusion matrix for multiclass classification. My objective is to learn the basic concept of constructing a confusion matrix for ...
- 11
1
vote
3
answers
143
views
Proof that Recall ≥ F1-score
I need to prove mathematically that the value of Recall is always bigger or equal to the F1-score.
The equation would be: $2 \times Precision \times \frac{Recall}{(Precision + Recall)} \le Recall$
...
- 13
2
votes
1
answer
179
views
How is it possible to get AUC = 0.72, but sensitivity = 0?
I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good:
However, using the same test data, this is the result of a ...
- 553
0
votes
0
answers
69
views
Cross-validation and F1 metric
Cross-validation with metrics such as F1 can be implemented in two ways:
For each cross-validation split, calculate F1_split on the validation dataset.
F1_result = average_by_splits(F1_split)
For ...
1
vote
1
answer
78
views
Metric to evaluate binary classification model on imbalanced dataset in order to meet percentage limitations
For a university class I'm working on a imbalanced dataset that has ratio of 43:1 Class_0 to Class_1.
Class_1 refer to companies that have declared bankruptcy based on feature columns of the dataset. ...
- 11
2
votes
2
answers
1k
views
Sensitivity vs. specificity vs. recall
Given a binary confusion matrix with true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), what are the formulas for sensitivity, specificity, and recall?
I'm ...
- 219
2
votes
2
answers
386
views
Is there a version of accuracy weighted by prevalence?
I have been learning the basic terminology for how to think about binary tests involving medical tests. The basic terms are here in this table
This is the confusion matrix.
My issue is the following. ...
- 4,453
5
votes
2
answers
638
views
Can I apply a confusion matrix to classification tasks outside of ML?
I would like to know if it's possible to use a confusion matrix to measure the performance of a classification tool outside the realm of ML or a statistical model.
For example, if I had a small script ...
- 51
5
votes
2
answers
1k
views
Poorly calibrated probabilities but good classification in confusion matrix
I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/...
- 275
1
vote
0
answers
38
views
Formula versus Non-Formula Interface Categorical Variables train() glmnet
I am comparing the confusion matrix between the formula interface and the non-formula interface using caret's train() for elastic net. I am trying to understand why the two interfaces produces ...
- 275
4
votes
1
answer
984
views
Why is "balanced accuracy" an arithmetic mean instead of harmonic?
In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
- 12.8k
1
vote
2
answers
802
views
How to analyse results of classification for time series + sliding windows
Here is my context:
I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows.
...
- 19
0
votes
1
answer
754
views
How to calculate the accuracy and error rate in confusion matrix?
(0 - negative outcome, 1 - positive outcome)
Am I correct? Because I found a different answer.
My answer:
Accuracy rate: TP+TN/ALL(58+25/58+44+23+25)=83/150
Error rate: FN+FP(23+44)= 67/150
Someone ...
- 21
0
votes
0
answers
33
views
When I convert the result of 'predict' function of logistic regression into factor, should the cutoff be same with the that of dependent variable? [duplicate]
I wanna run glm composed of dependent variable named "exposure", and independent variable named "counts" and "distance" respectively.
...
2
votes
1
answer
62
views
A metric for a big/medium/small ML classification
I am working on an ML classification task which is similar to the following:
Apples have to be classified to three classes: Big, Medium and Small.
I need a metric which I can use to assess the system. ...
- 123
0
votes
0
answers
36
views
When are we satisfied with the performance of a classifier?
I am training a multi-head classifier (based on Resnet18) for a multilabel classification task. The dataset I am using to train the classifier is noisy and to improve the accuracy for the worse-...
- 147
2
votes
0
answers
395
views
Are Type I and Type II errors in multiclass problem appear to be the same?
As far as I know from binary classification
FP error is a type 1 error
FN error is a type 2 error
I have this confusion matrix generated:
And here I found how to read this confusion matrix:
As you ...
- 121
1
vote
0
answers
101
views
How to assign costs to the confusion matrix
I am trying to assign costs to the confusion matrix. That is, in my problem, a false positive (FP) does not have the same cost as a false negative (FN), so I want to assign to these cases a cost "...
- 157
3
votes
1
answer
120
views
Two confusion matrices
In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to $...
- 1,706
0
votes
0
answers
41
views
Understand AUCs of different models
I'm testing two models against each other:
provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %.
provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
- 3,493
0
votes
0
answers
29
views
Make ROC graph with my problematic [duplicate]
For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic.
I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
- 1
1
vote
1
answer
57
views
How to interpret this confusion matrix and roc curve?
I got these two images for evaluating a RF:
I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %?
By looking only at ...
- 3,493
2
votes
2
answers
1k
views
Logistic Regression: What is the value for precision when recall (true positive rate) is 0?
A quick overview of definitions before I get into the question:
True Positive (TP): An actual positive that the model classified as positive
False Positive (FP): An actual negative that the model ...
- 719
1
vote
0
answers
314
views
Confusion matrix error [closed]
I have been trying to do support vector machine classification in R and confusion matrix has been showing the error even after changing into factors
confusionMatrix(as.factor(yTestPred), as.factor(...
- 11
3
votes
2
answers
4k
views
why I am getting TP and FP in confusion matrix is 0, how to get it right?
why I am getting TP and FP in confusion matrix is 0, how to get it right?
tp = 0,fp = 0,tn = 9847,fn = 18
1
vote
1
answer
40
views
Estimate sample size from a variable population
Context:
I want to measure accuracy, precision, and recall for individual raters. Each rater completes a variable amount of labels, for ex. rater A may complete 500 in a given time period while rater ...
