Questions tagged [confusion-matrix]
A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.
292
questions
34
votes
3
answers
80k
views
How can I interpret a confusion matrix
I am using confusion matrix to check the performance of my classifier.
I am using Scikit-Learn, I am little bit confused. How can I interpret the result from
...
- 1,157
34
votes
2
answers
29k
views
FPR (false positive rate) vs FDR (false discovery rate)
The following quote comes from the famous research paper Statistical significance for genome wide studies by Storey & Tibshirani (2003):
For example, a false positive rate of 5% means that on ...
- 405
26
votes
4
answers
165k
views
Given true positive, false negative rates, can you calculate false positive, true negative?
I have values for True Positive (TP) and False Negative (FN) as follows:
...
- 545
18
votes
3
answers
68k
views
How to build a confusion matrix for a multiclass classifier?
I have a problem with 6 classes. So I build a multiclass classifier, as follows: for each class, I have one Logistic Regression classifier, using One vs. All, which means that I have 6 different ...
- 333
18
votes
3
answers
10k
views
Relationship between the phi, Matthews and Pearson correlation coefficients
Are the phi and Matthews correlation coefficients the same concept? How are they related or equivalent to Pearson correlation coefficient for two binary variables? I assume the binary values are 0 and ...
- 19.5k
18
votes
1
answer
13k
views
Confidence interval of precision / recall and F1 score
To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to ...
- 537
16
votes
2
answers
47k
views
Using the caret package is it possible to obtain confusion matrices for specific threshold values?
I've obtained a logistic regression model (via train) for a binary response, and I've obtained the logistic confusion matrix via ...
- 356
13
votes
1
answer
20k
views
Is there any difference between Sensitivity and Recall?
In most of the places, I have found that sensitivity=recall. In terms of the Confusion Matrix, the formula for both of these is the same: $TP/(TP+FN)$.
Is there any difference between these two ...
11
votes
8
answers
8k
views
My machine learning model has precision of 30%. Can this model be useful?
I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
- 121
11
votes
3
answers
19k
views
What is no ' information rate ' algorithm?
I plan to implement ' no information rate ' as part of summary statistics. This statistic is implemented in r (Optimise SVM to avoid false-negative in binary classification) but not in Python (at ...
- 637
10
votes
3
answers
877
views
For a confusion matrix, is there a name for FP / (FP + FN)?
For a confusion matrix, there are a variety of useful rates, ratios and indices. But I cannot find the one I care about:
FP / (FP + FN)
Of course this measure is ...
- 203
9
votes
1
answer
2k
views
What does it imply when the sensitivity = 1.000 and specificity = 0.000?
What adjustments do I need to make when I have extreme values in the confusion matrix as stated above i.e
sensitivity = 1 and specificity = 0?
- 287
9
votes
1
answer
4k
views
Can F1-Score be higher than accuracy?
I'm using sklearn's confusion_matrix and classification_report methods to compute the ...
- 369
8
votes
2
answers
386
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
- 124k
7
votes
2
answers
8k
views
How to threshold multiclass probability prediction to get confusion matrix?
Lets say my multinomial logistic regression predict that a chance of a sample belonging to a each class is A=0.6, B=0.3, C=0.1 How do I threshold this values to get just binary prediction of a sample ...
- 7,609
7
votes
3
answers
10k
views
How to compare 2 classifers using confusion matrix?
How to compare 2 classifiers using Confusion Matrix?
For example, if we have 2 confusion matrix(binary classification) obtained from different classifiers or using different features, how I can ...
- 2,207
7
votes
2
answers
860
views
Is there something like a confusion matrix for a probabilistic score rather than categories?
Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses.
But then we ...
- 62.4k
7
votes
3
answers
8k
views
How can I derive confidence intervals from the confusion matrix for a classifier?
I am using k-fold cross-validation to generate a confusion matrix for a classifier. I need to calculate 95% confidence intervals for the number of times each class is predicted when run against a ...
- 171
6
votes
5
answers
14k
views
Are FAR and FRR the same as FPR and FNR, respectively?
FPR = False Positive Rate
FNR = False Negative Rate
FAR = False Acceptance Rate
FRR = False Rejection Rate
Are they the same? if Not, is it possible to calculate FAR and FRR from the confusion ...
- 1,179
6
votes
1
answer
10k
views
Definition of p-value in carets confusion matrix method
In the documentation for the confusion matrix method in the caret package, the p-value is described as:
a one-sided test to see if the accuracy is better than the "no information
rate," which is ...
5
votes
3
answers
50k
views
How to understand confusion matrix for 3x3
I used Sklearn logistic regression for multiclass classifier to classify as Male , Female and Infant on abalone data set
Below is my sample Logistic regression for multi classifier
...
- 225
5
votes
2
answers
578
views
Can I apply a confusion matrix to classification tasks outside of ML?
I would like to know if it's possible to use a confusion matrix to measure the performance of a classification tool outside the realm of ML or a statistical model.
For example, if I had a small script ...
- 51
5
votes
1
answer
82
views
Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)
This answer to a question on Math Stack Exchange got me thinking about a confusion matrix as more than just a rectangular array of numbers. We don’t talk about a confusion matrix as a linear ...
- 62.4k
5
votes
1
answer
270
views
Is ROC curve unique?
ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter ...
- 3,933
5
votes
2
answers
744
views
Poorly calibrated probabilities but good classification in confusion matrix
I have an imbalanced data set. My goal is to balance sensitivity and specificity via the confusion matrix. I used glmnet in r with class weights. The model does well at balancing the sensitivity/...
- 265
5
votes
1
answer
2k
views
Confusion matrix, metrics, & joint vs. conditional probabilities
In the binary classification/prediction problem we have unknown labels $y\in\{0,1\}$, which we try to predict using an estimator $\hat{y}$. Commonly the performance of an estimator is summarized using ...
- 13k
5
votes
2
answers
3k
views
Calculate classifier accuracy from per label accuracy
I would like to have a per Label accuracy and classifier level accuracy, but my calculations seem incorrect. Here is my full example.
Let's say I have a multilabel classifier which predicted in the ...
- 137
5
votes
1
answer
40k
views
Confusion matrix and ROC curves
I have built a model to predict Upsell probability. When I use the function confusionMatirx from caret package, I get the following results:
...
- 91
5
votes
1
answer
147
views
Can Accuracy be higher than both sensitivity and specificity?
I came across a paper which reported the following results
Accuracy
Specificity
Sensitivity
97.49%
93.6%
94.3%
It seems unusual for accuracy to be higher than both sensitivity and specificity. Is ...
- 163
5
votes
1
answer
280
views
Searching for appropriate binary classification metric
Problem:
I am working on a a binary classification problem with the following qualities:
Highly imbalanced
False Negative errors much more costly than False Positive
False Positives are only False ...
- 303
5
votes
2
answers
3k
views
How to make a confusion matrix from comparing prediction results of two algorithms?
I applied two unsupervised algorithms to the same data, and would like to make a confusion matrix out of results, how should I achieve it in R?
An example with R codes like following:
...
- 667
5
votes
0
answers
6k
views
Confusion matrix for multilabel classification
I know that a similar subject was treated here, but my question is a little bit different.
I have a result of multilabel classification, like this (2 observations, 3 labels in the example, in ...
- 81
5
votes
0
answers
570
views
How to interpret the P-value and CI for confusionMatrix output in R caret? [duplicate]
A typical confusion matrix from R's caret package might look like this:
...
- 73
5
votes
0
answers
1k
views
Confusion matrix with multi-class multi-label classification
Let's say I have three possible classes {'isCold' 'isWet' 'isSolid'} and my instances can belong to one or more of these classes.
Ground Truth
ice = {'isCold' 'isWet' 'isSolid'}
water = {'isCold' '...
- 301
4
votes
2
answers
3k
views
Is there a name for this metric: TN / (TN + FN)?
Given a confusion matrix, there's all kind of metrics: Accuracy, Precision, Recall/Sensitivity, Specificity.
But I haven't seen any name for the ratio between the TN (True Negative) and the sum of ...
- 3,413
4
votes
2
answers
9k
views
How to calculate information included in R's confusion matrix
I want know the formulas for the information highlighted in the confusion matrix in the picture below:
- 141
4
votes
2
answers
6k
views
Which quality measures are available for non-binary problems?
I'm wondering which quality measures are available for non-binary classifiers. I've read this article
https://en.wikipedia.org/wiki/Receiver_operating_characteristic
I understand, that the idea of ...
- 205
4
votes
1
answer
1k
views
How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?
How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?
Since in a binary classifier, there are two classes, often ...
- 305
4
votes
2
answers
18k
views
How to find true positive, true negative, false positive, false negative from a three class confusion matrix?
I built a confusion matrix of three class. like, (a,b,c are the class)
- a b c <= predicted
a 20 5 0
b 7 18 0
c 0 0 20
Now I want to ...
- 176
4
votes
1
answer
14k
views
Interpretation of a WEKA result buffer - confusion matrix and performance
I want to know how to get several performance measurements of a generated WEKA model. Note that I am predicting a two-class variable, Alive or ...
4
votes
1
answer
2k
views
How to interpret/deal with low precision confusion matrix?
I am developing a fraud detection model with unbalanced data set ,
I used Random Forests Algorithm.
The confusion matrix showed that model has good sensitivity(recall) but low precision(Pos Pred ...
4
votes
1
answer
13k
views
How to find TP,TN, FP and FN values from 8x8 Confusion Matrix
I have confusion matrix as follow:
a b c d e f g h <-- classified as
1086 7 1 0 2 4 0 0 | a
7 1064 8 6 0 2 2 0 | b
0 ...
- 143
4
votes
1
answer
2k
views
Confusion about confusion matrix, or why 'power of test' + $\beta$ = 1?
Consider a confusion table summarizing the outcomes of $n$ tests of hypothesis $H_0$ for $n$ independent experiments:
...
- 2,634
4
votes
1
answer
240
views
One or two-sided test for classifier accuracy?
As far as I could reconstruct, caret::confusionMatrix uses a one-sided binomial test to compute the p-value of the accuracy being better than the "no ...
- 9,139
4
votes
1
answer
151
views
Accuracy assessment for multinomial probabilities
I am working on a classification problem where the input data are multinational proportions for each point, with $m$>2 groups. And the outcome is likewise a multinomial proportion for that point. ...
- 271
4
votes
0
answers
89
views
Why is information gain rarely used as a reported effect size?
Commonly, in reporting effect sizes in medical articles, papers typically report: risk difference (RD), relative risk (RR) and odds ratio (OR) (and for variable timescale cases and other situations, ...
- 605
4
votes
0
answers
205
views
False positives and False negatives of three or more classes
I have a $10\times 10$ confusion matrix $M$ generated after to execute an KNN classification process for digits recognition (0,1,2...9).
As usual, each row of $M$ represent the "true/real" class of ...
- 41
3
votes
1
answer
9k
views
The relation between a confusion matrix and a ROC curve
As an example I have a confusion matrix that shows good accuracy but poor performance on sensitivity because of imbalanced classes. I made this fictive table for a presentation.
...
- 510
3
votes
3
answers
2k
views
How to visualize classifier output?
Problem description: I have an Excel sheet containing a table of true and estimated values. I want to somehow capture this information over a plot of actual (horizontal axis) vs estimated values (...
- 131
3
votes
1
answer
20k
views
Random Forest confusion matrix
I've been creating some random forest models using the caret package in R. I don't have a large amount of data to work with so I'm using 10 x 10-fold CV in lieu of an independent test set.
When I ...
- 261