# Questions tagged [confusion-matrix]

A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.

297 questions
Filter by
Sorted by
Tagged with
57 views

### False Negative vs False Positive for Multiclass classification

Suppose I have three classes 1,2,3. And there's evaluation like below, where second element is false prediction where model predict class 3 while ground truth is 2. ...
503 views

### Judging a model through the TP, TN, FP, and FN values

I am evaluating a model that predicts the existence or not existence of a "characteristic" (for example, "there is a dog in this image") using several datasets. The system outputs ...
• 247
135 views

### Using Predictive Value Confidence Intervals to "Predict" Outcomes

Here's the quick version: Say I have a confusion matrix with the following data based on a proficiency cut score on a pretest and outcomes on (passing/failing) a class. The cut score was determined ...
57 views

### How to interpret this confusion matrix in R?

I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%. These are the results of my confusion matrix: Confusion Matrix and Statistics Confusion Matrix (...
• 41
7 views

### Is the confusion matrix meaning the same within deep learning than within hypothesis testing?

Is the confusion matrix meaning the same within deep learning than within hypothesis testing? I've read what the matrix is, and how to get the values. But in the case of deep learning for ...
• 174
69 views

### Is relying on just the confusion matrix for highly imbalanced test sets to evaluate model performance a bad idea?

I have a binary classification model with a test set that is highly skewed, the majority class 0 is 22 times greater than the minority class 1. This causes my Precision to be low and Recall to be high,...
1 vote
37 views

### Help interpreting multi-class confusion matrix

I'm looking at the SAMHSA Mental Health Client-Level dataset. I did some t-SNE plots (dropping irrelevant cols, normalizing some, one-hot encoding some) of 500k rows out of 6.5mil. I'm trying to do ...
108 views

### Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)

This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
• 65.1k
39 views

### Precision and recall in the case of null predictive power

With a multiclass classification problem, if it is perfectly predictive, the probability, $P(j|i)$, of predicting class $j$ for the real class $i$ is 1 when $j=1$ and 0 otherwhise. In this case the ...
• 141
63 views

### Matthews' Correlation Coefficient and Leave-One-Out Cross Validation

How do I determine Matthews' Correlation Coefficient for leave-one-out cross-validation? There doesn't seem to be a way to average the coefficient across folds - one element is insufficient support ...
45 views

### Drawing a dendrogram based on a confusion matrix

I have a simple 2D confusion matrix that I normalised for every true class. Now, what would be the best method to visualise this as a dendrogram? I have seen hierarchical agglomerative clustering on ...
17 views

### How do you calculate multiclass microaverage precision and recall?

I have a confusion matrix that is 4 class. ...
279 views

### Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
• 4,429
884 views

### Imbalanced classes and possible ways to increase precision, recall and f1-score of the prediction model

I've just started my data science internship, and this is my first time in the field. I'm sure I'll face challenges in the future where I might need your help. It's also my first time asking a ...
76 views

### What should I do when a test specificity is zero?

Upon doing a test that compares the effect of imaging versus regular tissue biopsy for cancer that has a prevalence of 90%, we ...
917 views

### For a confusion matrix, is there a name for FP / (FP + FN)?

For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about: FP / (FP + FN) Of course this measure is ...
1 vote
16 views

### Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?

Given the following table of predictions vs. actual states: ...
• 1,109
106 views

### Overfitting models in mlr3

I'm trying to compare multiple learners on my dataset (called "data") in order to predict a target called "lesionResponse", with custom resampling. Since mlr3 package doesn't allow ...
• 13
268 views

### Customized F1-Score for multi-class classification

Let's consider a multi-class classification problem with 4 classes: 0, 1, 2, and 3 F1-Score 'macro'-averaged is calculated like that: ...
• 127
119 views

### Confidence Informed Confusion Matrix (Threshold Free)

I am using the normalized confusion matrix to aid in quantifying the uncertainty in related observations over time. More specifically, I have a classifier that returns the confidence of each class via ...
• 31
1 vote
97 views

### Surprising disparity between Confusion matrix values and AUC?

I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
230 views

### Can Accuracy be higher than both sensitivity and specificity?

I came across a paper which reported the following results Accuracy Specificity Sensitivity 97.49% 93.6% 94.3% It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
• 163
8k views

### My machine learning model has precision of 30%. Can this model be useful?

I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
• 121
56 views

### Confusion Matrix with a class created from lower probabilities of other classes

Please comment on the following method: Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here. Then ...
• 287
430 views

### Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
• 128k
1 vote
52 views

### How to measure sensitivity, speficity, PPV and NPV from confusion matrix for multiclass classification [duplicate]

I want to calculate sensitivity, specificity, PPV, and NPV from a confusion matrix for multiclass classification. My objective is to learn the basic concept of constructing a confusion matrix for ...
• 11
1 vote
130 views

### Proof that Recall ≥ F1-score

I need to prove mathematically that the value of Recall is always bigger or equal to the F1-score. The equation would be: $2 \times Precision \times \frac{Recall}{(Precision + Recall)} \le Recall$ ...
• 13
167 views

### How is it possible to get AUC = 0.72, but sensitivity = 0?

I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good: However, using the same test data, this is the result of a ...
• 543
59 views

### Cross-validation and F1 metric

Cross-validation with metrics such as F1 can be implemented in two ways: For each cross-validation split, calculate F1_split on the validation dataset. F1_result = average_by_splits(F1_split) For ...
1 vote
64 views

### Metric to evaluate binary classification model on imbalanced dataset in order to meet percentage limitations

For a university class I'm working on a imbalanced dataset that has ratio of 43:1 Class_0 to Class_1. Class_1 refer to companies that have declared bankruptcy based on feature columns of the dataset. ...
• 11
894 views

### Sensitivity vs. specificity vs. recall

Given a binary confusion matrix with true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), what are the formulas for sensitivity, specificity, and recall? I'm ...
• 219
306 views

### Is there a version of accuracy weighted by prevalence?

I have been learning the basic terminology for how to think about binary tests involving medical tests. The basic terms are here in this table This is the confusion matrix. My issue is the following. ...
• 4,423
598 views

### Can I apply a confusion matrix to classification tasks outside of ML?

I would like to know if it's possible to use a confusion matrix to measure the performance of a classification tool outside the realm of ML or a statistical model. For example, if I had a small script ...
989 views

### Poorly calibrated probabilities but good classification in confusion matrix

I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/...
• 275
1 vote
38 views

### Formula versus Non-Formula Interface Categorical Variables train() glmnet

I am comparing the confusion matrix between the formula interface and the non-formula interface using caret's train() for elastic net. I am trying to understand why the two interfaces produces ...
• 275
864 views

### Why is "balanced accuracy" an arithmetic mean instead of harmonic?

In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
• 12.8k
1 vote
701 views

### How to analyse results of classification for time series + sliding windows

Here is my context: I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows. ...
• 19
700 views

### How to calculate the accuracy and error rate in confusion matrix?

(0 - negative outcome, 1 - positive outcome) Am I correct? Because I found a different answer. My answer: Accuracy rate: TP+TN/ALL(58+25/58+44+23+25)=83/150 Error rate: FN+FP(23+44)= 67/150 Someone ...
33 views

### When I convert the result of 'predict' function of logistic regression into factor, should the cutoff be same with the that of dependent variable? [duplicate]

I wanna run glm composed of dependent variable named "exposure", and independent variable named "counts" and "distance" respectively. ...
57 views

### A metric for a big/medium/small ML classification

I am working on an ML classification task which is similar to the following: Apples have to be classified to three classes: Big, Medium and Small. I need a metric which I can use to assess the system. ...
• 123
35 views

### When are we satisfied with the performance of a classifier?

I am training a multi-head classifier (based on Resnet18) for a multilabel classification task. The dataset I am using to train the classifier is noisy and to improve the accuracy for the worse-...
• 137
344 views

### Are Type I and Type II errors in multiclass problem appear to be the same?

As far as I know from binary classification FP error is a type 1 error FN error is a type 2 error I have this confusion matrix generated: And here I found how to read this confusion matrix: As you ...
• 121
1 vote
92 views

### How to assign costs to the confusion matrix

I am trying to assign costs to the confusion matrix. That is, in my problem, a false positive (FP) does not have the same cost as a false negative (FN), so I want to assign to these cases a cost "...
• 157
113 views

### Two confusion matrices

In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to \$...
• 1,676
40 views

### Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
• 3,533
29 views

### Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
1 vote
56 views

### How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
• 3,533
1k views

### Logistic Regression: What is the value for precision when recall (true positive rate) is 0?

A quick overview of definitions before I get into the question: True Positive (TP): An actual positive that the model classified as positive False Positive (FP): An actual negative that the model ...
• 719
1 vote