Skip to main content

Questions tagged [confusion-matrix]

A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.

Filter by
Sorted by
Tagged with
0 votes
3 answers
57 views

False Negative vs False Positive for Multiclass classification

Suppose I have three classes 1,2,3. And there's evaluation like below, where second element is false prediction where model predict class 3 while ground truth is 2. ...
Muhammad Ikhwan Perwira's user avatar
4 votes
3 answers
503 views

Judging a model through the TP, TN, FP, and FN values

I am evaluating a model that predicts the existence or not existence of a "characteristic" (for example, "there is a dog in this image") using several datasets. The system outputs ...
KansaiRobot's user avatar
0 votes
0 answers
135 views

Using Predictive Value Confidence Intervals to "Predict" Outcomes

Here's the quick version: Say I have a confusion matrix with the following data based on a proficiency cut score on a pretest and outcomes on (passing/failing) a class. The cut score was determined ...
jacev123's user avatar
0 votes
1 answer
57 views

How to interpret this confusion matrix in R?

I've produced a logistic regression model using 70% of my data and tested it using the remaining 30%. These are the results of my confusion matrix: Confusion Matrix and Statistics Confusion Matrix (...
Laura's user avatar
  • 41
0 votes
0 answers
7 views

Is the confusion matrix meaning the same within deep learning than within hypothesis testing?

Is the confusion matrix meaning the same within deep learning than within hypothesis testing? I've read what the matrix is, and how to get the values. But in the case of deep learning for ...
Mah Neh's user avatar
  • 174
2 votes
3 answers
69 views

Is relying on just the confusion matrix for highly imbalanced test sets to evaluate model performance a bad idea?

I have a binary classification model with a test set that is highly skewed, the majority class 0 is 22 times greater than the minority class 1. This causes my Precision to be low and Recall to be high,...
statsnoob's user avatar
1 vote
0 answers
37 views

Help interpreting multi-class confusion matrix

I'm looking at the SAMHSA Mental Health Client-Level dataset. I did some t-SNE plots (dropping irrelevant cols, normalizing some, one-hot encoding some) of 500k rows out of 6.5mil. I'm trying to do ...
Jackson Walters's user avatar
5 votes
1 answer
108 views

Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)

This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
Dave's user avatar
  • 65.1k
3 votes
1 answer
39 views

Precision and recall in the case of null predictive power

With a multiclass classification problem, if it is perfectly predictive, the probability, $P(j|i)$, of predicting class $j$ for the real class $i$ is 1 when $j=1$ and 0 otherwhise. In this case the ...
arivero's user avatar
  • 141
0 votes
1 answer
63 views

Matthews' Correlation Coefficient and Leave-One-Out Cross Validation

How do I determine Matthews' Correlation Coefficient for leave-one-out cross-validation? There doesn't seem to be a way to average the coefficient across folds - one element is insufficient support ...
Chris Coffee's user avatar
0 votes
0 answers
45 views

Drawing a dendrogram based on a confusion matrix

I have a simple 2D confusion matrix that I normalised for every true class. Now, what would be the best method to visualise this as a dendrogram? I have seen hierarchical agglomerative clustering on ...
Damiaan Reijnaers's user avatar
0 votes
0 answers
17 views

How do you calculate multiclass microaverage precision and recall?

I have a confusion matrix that is 4 class. ...
Captain Toad's user avatar
5 votes
1 answer
279 views

Is ROC curve unique?

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
Roger V.'s user avatar
  • 4,429
2 votes
1 answer
884 views

Imbalanced classes and possible ways to increase precision, recall and f1-score of the prediction model

I've just started my data science internship, and this is my first time in the field. I'm sure I'll face challenges in the future where I might need your help. It's also my first time asking a ...
Thimali Fernando's user avatar
0 votes
0 answers
76 views

What should I do when a test specificity is zero?

Upon doing a test that compares the effect of imaging versus regular tissue biopsy for cancer that has a prevalence of 90%, we ...
Mohamed Rahouma's user avatar
10 votes
3 answers
917 views

For a confusion matrix, is there a name for FP / (FP + FN)?

For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about: FP / (FP + FN) Of course this measure is ...
David Bridgeland's user avatar
1 vote
1 answer
16 views

Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?

Given the following table of predictions vs. actual states: ...
Bryan's user avatar
  • 1,109
0 votes
0 answers
106 views

Overfitting models in mlr3

I'm trying to compare multiple learners on my dataset (called "data") in order to predict a target called "lesionResponse", with custom resampling. Since mlr3 package doesn't allow ...
Nicolas's user avatar
  • 13
2 votes
1 answer
268 views

Customized F1-Score for multi-class classification

Let's consider a multi-class classification problem with 4 classes: 0, 1, 2, and 3 F1-Score 'macro'-averaged is calculated like that: ...
AngelMarcos's user avatar
3 votes
0 answers
119 views

Confidence Informed Confusion Matrix (Threshold Free)

I am using the normalized confusion matrix to aid in quantifying the uncertainty in related observations over time. More specifically, I have a classifier that returns the confidence of each class via ...
Jared's user avatar
  • 31
1 vote
1 answer
97 views

Surprising disparity between Confusion matrix values and AUC?

I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
AnimNeuroGrad's user avatar
5 votes
1 answer
230 views

Can Accuracy be higher than both sensitivity and specificity?

I came across a paper which reported the following results Accuracy Specificity Sensitivity 97.49% 93.6% 94.3% It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
Jack O'Neill's user avatar
11 votes
8 answers
8k views

My machine learning model has precision of 30%. Can this model be useful?

I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
wmmwmm's user avatar
  • 121
0 votes
1 answer
56 views

Confusion Matrix with a class created from lower probabilities of other classes

Please comment on the following method: Model was trained on data labeled with 4 classes: header, question, answer an other. Predictions were done on a separate test set, nothing new here. Then ...
dokondr's user avatar
  • 287
8 votes
2 answers
430 views

Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
Stephan Kolassa's user avatar
1 vote
0 answers
52 views

How to measure sensitivity, speficity, PPV and NPV from confusion matrix for multiclass classification [duplicate]

I want to calculate sensitivity, specificity, PPV, and NPV from a confusion matrix for multiclass classification. My objective is to learn the basic concept of constructing a confusion matrix for ...
SUZ's user avatar
  • 11
1 vote
3 answers
130 views

Proof that Recall ≥ F1-score

I need to prove mathematically that the value of Recall is always bigger or equal to the F1-score. The equation would be: $2 \times Precision \times \frac{Recall}{(Precision + Recall)} \le Recall$ ...
Agi's user avatar
  • 13
2 votes
1 answer
167 views

How is it possible to get AUC = 0.72, but sensitivity = 0?

I've trained a Gradient Boost classifier and used it to predict a binary target variable. I plot the ROC, here it is, looks nice and good: However, using the same test data, this is the result of a ...
CORy's user avatar
  • 543
0 votes
0 answers
59 views

Cross-validation and F1 metric

Cross-validation with metrics such as F1 can be implemented in two ways: For each cross-validation split, calculate F1_split on the validation dataset. F1_result = average_by_splits(F1_split) For ...
Arseniy Maryin's user avatar
1 vote
1 answer
64 views

Metric to evaluate binary classification model on imbalanced dataset in order to meet percentage limitations

For a university class I'm working on a imbalanced dataset that has ratio of 43:1 Class_0 to Class_1. Class_1 refer to companies that have declared bankruptcy based on feature columns of the dataset. ...
pchi's user avatar
  • 11
2 votes
2 answers
894 views

Sensitivity vs. specificity vs. recall

Given a binary confusion matrix with true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), what are the formulas for sensitivity, specificity, and recall? I'm ...
user72716's user avatar
  • 219
2 votes
2 answers
306 views

Is there a version of accuracy weighted by prevalence?

I have been learning the basic terminology for how to think about binary tests involving medical tests. The basic terms are here in this table This is the confusion matrix. My issue is the following. ...
Stan Shunpike's user avatar
5 votes
2 answers
598 views

Can I apply a confusion matrix to classification tasks outside of ML?

I would like to know if it's possible to use a confusion matrix to measure the performance of a classification tool outside the realm of ML or a statistical model. For example, if I had a small script ...
user370003's user avatar
5 votes
2 answers
989 views

Poorly calibrated probabilities but good classification in confusion matrix

I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/...
mapleleaf's user avatar
  • 275
1 vote
0 answers
38 views

Formula versus Non-Formula Interface Categorical Variables train() glmnet

I am comparing the confusion matrix between the formula interface and the non-formula interface using caret's train() for elastic net. I am trying to understand why the two interfaces produces ...
mapleleaf's user avatar
  • 275
4 votes
1 answer
864 views

Why is "balanced accuracy" an arithmetic mean instead of harmonic?

In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
shadowtalker's user avatar
  • 12.8k
1 vote
2 answers
701 views

How to analyse results of classification for time series + sliding windows

Here is my context: I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows. ...
Adrien's user avatar
  • 19
0 votes
1 answer
700 views

How to calculate the accuracy and error rate in confusion matrix?

(0 - negative outcome, 1 - positive outcome) Am I correct? Because I found a different answer. My answer: Accuracy rate: TP+TN/ALL(58+25/58+44+23+25)=83/150 Error rate: FN+FP(23+44)= 67/150 Someone ...
Woddyopened's user avatar
0 votes
0 answers
33 views

When I convert the result of 'predict' function of logistic regression into factor, should the cutoff be same with the that of dependent variable? [duplicate]

I wanna run glm composed of dependent variable named "exposure", and independent variable named "counts" and "distance" respectively. ...
user364400's user avatar
2 votes
1 answer
57 views

A metric for a big/medium/small ML classification

I am working on an ML classification task which is similar to the following: Apples have to be classified to three classes: Big, Medium and Small. I need a metric which I can use to assess the system. ...
Alexey's user avatar
  • 123
0 votes
0 answers
35 views

When are we satisfied with the performance of a classifier?

I am training a multi-head classifier (based on Resnet18) for a multilabel classification task. The dataset I am using to train the classifier is noisy and to improve the accuracy for the worse-...
randomal's user avatar
  • 137
2 votes
0 answers
344 views

Are Type I and Type II errors in multiclass problem appear to be the same?

As far as I know from binary classification FP error is a type 1 error FN error is a type 2 error I have this confusion matrix generated: And here I found how to read this confusion matrix: As you ...
Elvin's user avatar
  • 121
1 vote
0 answers
92 views

How to assign costs to the confusion matrix

I am trying to assign costs to the confusion matrix. That is, in my problem, a false positive (FP) does not have the same cost as a false negative (FN), so I want to assign to these cases a cost "...
PicaPython's user avatar
3 votes
1 answer
113 views

Two confusion matrices

In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to $...
Frank's user avatar
  • 1,676
0 votes
0 answers
40 views

Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
Ben's user avatar
  • 3,533
0 votes
0 answers
29 views

Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
yonafunu's user avatar
1 vote
1 answer
56 views

How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
Ben's user avatar
  • 3,533
2 votes
2 answers
1k views

Logistic Regression: What is the value for precision when recall (true positive rate) is 0?

A quick overview of definitions before I get into the question: True Positive (TP): An actual positive that the model classified as positive False Positive (FP): An actual negative that the model ...
Nova's user avatar
  • 719
1 vote
0 answers
305 views

Confusion matrix error [closed]

I have been trying to do support vector machine classification in R and confusion matrix has been showing the error even after changing into factors confusionMatrix(as.factor(yTestPred), as.factor(...
Brintha's user avatar
  • 11
3 votes
2 answers
4k views

why I am getting TP and FP in confusion matrix is 0, how to get it right?

why I am getting TP and FP in confusion matrix is 0, how to get it right? tp = 0,fp = 0,tn = 9847,fn = 18
Abhishek Pandey's user avatar

1
2 3 4 5 6