56
votes
Accepted
FPR (false positive rate) vs FDR (false discovery rate)
I'm going to explain these in a few different ways because it helped me understand it.
Let's take a specific example. You are doing a test for a disease on a group of people. Now let's define some ...
- 20.4k
29
votes
Accepted
My machine learning model has precision of 30%. Can this model be useful?
As Dave argues, if "false negatives have no associated costs", then your best course of action would be not to classify anything as positive, i.e., as 1. You inspect nothing at all, you ...
- 131k
24
votes
Accepted
Is there any difference between Sensitivity and Recall?
It is not uncommon that statistical tools have different origins and names, but same meaning.
The name sensitivity comes from the statistics domain as a measure for the performance of a binary ...
- 656
17
votes
My machine learning model has precision of 30%. Can this model be useful?
I would say neither group is entirely correct. The question is what do you want to do with the model, and what will happen for positive or negative model predictions?
There are screening tests used ...
- 46.8k
15
votes
How to build a confusion matrix for a multiclass classifier?
While there are some answers already on this forum I thought I'd give the explicit equations to make it more definite:
Assuming you have a multi-class confusion matrix of the form,
\begin{align}
C=\...
- 263
14
votes
Accepted
What does it imply when the sensitivity = 1.000 and specificity = 0.000?
Sensitivity $= 1$ means you had some true positives and no false negatives: all actual cases were correctly predicted as positive
Specificity $= 0$ means you had some false positives and no true ...
- 42.2k
12
votes
Accepted
How to understand confusion matrix for 3x3
Based on the 3x3 confusion matrix in your example (assuming I'm understanding the labels correctly) the columns are the predictions and the rows must therefore be the actual values. The main diagonal (...
- 707
10
votes
Accepted
What is no ' information rate ' algorithm?
Suppose that you have response $y_i$ and covariates $x_i$ for $i = 1 ...n$, and some loss function $\mathcal{L}$. The no information error rate of a model $f$ is the average loss of $f$ over all ...
- 496
10
votes
Accepted
8
votes
How to threshold multiclass probability prediction to get confusion matrix?
According to @cangrejo's answer: https://stats.stackexchange.com/a/310956/194535, suppose the original output probability of your model is the vector $v$, and then you can define the prior ...
- 181
8
votes
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Short answer
You can't.
Somewhat longer answer
The Brier score or log score are calculated from probabilistic classifications and corresponding outcomes. The confusion matrix, accuracy etc. are ...
- 131k
8
votes
My machine learning model has precision of 30%. Can this model be useful?
This primarily depends on how the model is supposed to be used. From your context it seems you have an alternative test which has an almost perfect classification rate but is very expensive to use ...
- 498
8
votes
For a confusion matrix, is there a name for FP / (FP + FN)?
I never heard the name for it and Wikipedia lists most of the named metrics like this, so my guess would be that it does not have a name.
- 141k
8
votes
Accepted
For a confusion matrix, is there a name for FP / (FP + FN)?
I would call this the proportion of the misclassifications/mistakes that are false positives.
The denominator is the total number of misclassifications/mistakes. Of these mistakes, some are false ...
- 67.2k
8
votes
Is ROC curve unique?
Maybe I don't understand your notation, but I feel you are mixing up true probabilities, model outputs (which might be probability estimates) and events.
given an event E, model output $f(\theta)$ (...
- 7,262
7
votes
Accepted
7
votes
How to understand confusion matrix for 3x3
True Positive, False Positive and similar counts and rates only make sense if there is a notion of "positive" and "negative" classes in your data. That is, only if you have exactly two classes. You ...
- 131k
7
votes
Accepted
Can F1-Score be higher than accuracy?
This is definitely possible, and not strange at all.
Recall how accuracy and the F1 score are defined:
$$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\quad\text{and}\quad \text{F1}=\frac{2TP}{2TP+FP+FN}. $...
- 131k
7
votes
My machine learning model has precision of 30%. Can this model be useful?
As with most things, neither of two polar opinions is wholly correct
If a model can select an area needed for inspection better than random chance then for a singular inspection run you're using your ...
- 225
7
votes
Linear algebra properties of a confusion matrix (eigenvalues, eigenvectors, and determinants)
The eigenvalues would really only reveal how many classes (single classifier) or how many classifiers are correlated with one another (multiple classifiers). But if you look at the quasi-diagonalized ...
- 870
7
votes
Judging a model through the TP, TN, FP, and FN values
Do not use any of accuracy, precision, recall, or the F1 score. They all suffer from the same issues, especially - but not only - for "unbalanced" data: Why is accuracy not the best measure ...
- 131k
6
votes
Confusion matrix, metrics, & joint vs. conditional probabilities
For predicted labels $\hat{y}$ and true labels $y\in\{0,1\}$, the confusion matrix is given by
\begin{array}{c|c:c|c}
& y=0 & y=1 & \\
\hline
\hat{y}=0 & \mathrm{TN} & \mathrm{FN}...
- 13.1k
5
votes
Accepted
How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?
In my experience, yes, it does.
Some libraries may give you a weighted average of the AUPRC across all of the classes, in addition to . For example, Weka includes a class that has a function to ...
- 3,086
5
votes
Confidence interval of precision / recall and F1 score
To give some quick answers to the points raised:
The additional "$+4$"observed when calculated the "adjusted version of recall". This comes from the viewing the occurrence of a True Positive as a ...
- 46.1k
5
votes
Formula for expected false positive & negative rates in hiring decisions based on r
Let's assume that it is useful to dichotomize job performance after hiring.
That is a strong assumption. But let's go with it.
Let $X$ denote the predictor and $Y$ the actual performance. Let's ...
- 131k
5
votes
Accepted
Why is "balanced accuracy" an arithmetic mean instead of harmonic?
For typical accuracy, we collect all the correct results in the numerator and divide it by the total number of samples. This doesn't account for class imbalance. If we balance the classes by giving ...
- 58.2k
5
votes
Poorly calibrated probabilities but good classification in confusion matrix
This topic has been widely discussed, especially in some answers by Stephan Kolassa. I will try to summarize the main take-home messages for your specific question.
From a pure statistical point of ...
- 51
5
votes
Can I apply a confusion matrix to classification tasks outside of ML?
Sure you can. Those metrics are older than machine learning. For example, the ROC curves calculated based on TPR and FPR were designed during World War II for judging the accuracy of radars. The ...
- 141k
4
votes
How do you calculate cost of a confusion matrix with more than two classes
To obtain the cost you simply have to multiply each term in your confusion matrix by its cost and then sum the terms.
if your confusion matrix is:
$$
\begin{matrix}
23 & 4 & 0\\
6 & 13 &...
- 99
4
votes
How to build a confusion matrix for a multiclass classifier?
Using the matrix attached in the question and considering the values in the vertical axis as the actual class, and the values in the horizontal axis the prediction. Then for the Class 1:
True ...
- 207
Only top scored, non community-wiki answers of a minimum length are eligible