76

In a 2-hypothesis case, the confusion matrix is usually: Declare H1 Declare H0 Is H1 TP FN Is H0 FP TN where I've used something similar to your notation: TP = true positive (declare H1 when, in truth, H1), FN = false negative (declare H0 when, in truth, H1), FP = false positive TN = true negative From the raw data, the values in the table would ...


64

You're on the right track. So a few things right off the bat. From the definition of the two metrics, we have that IoU and F score are always within a factor of 2 of each other: $$ F/2 \leq IoU \leq F $$ and also that they meet at the extremes of one and zero under the conditions that you would expect (perfect match and completely disjoint). Note also that ...


59

I cannot think of an intuitive meaning of the F measure, because it's just a combined metric. What's more intuitive than F-mesure, of course, is precision and recall. But using two values, we often cannot determine if one algorithm is superior to another. For example, if one algorithm has higher precision but lower recall than other, how can you tell which ...


33

Here are the conclusions from a paper by Davis & Goadrich explaining the relationship between ROC and PR space. They answer the first two questions: First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it ...


32

Good summary paper, looking at these metrics for multi-class problems: Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45, p. 427-437. (pdf) The abstract reads: This paper presents a systematic analysis of twenty four performance measures ...


32

The question is quite vague so I am going to assume you want to choose an appropriate performance measure to compare different models. For a good overview of the key differences between ROC and PR curves, you can refer to the following paper: The Relationship Between Precision-Recall and ROC Curves by Davis and Goadrich. To quote Davis and Goadrich: ...


30

The f1-score gives you the harmonic mean of precision and recall. The scores corresponding to every class will tell you the accuracy of the classifier in classifying the data points in that particular class compared to all other classes. The support is the number of samples of the true response that lie in that class. You can find documentation on both ...


29

The importance of the F1 score is different based on the scenario. Lets assume the target variable is a binary label. Balanced class: In this situation, the F1 score can effectively be ignored, the mis-classification rate is key. Unbalanced class, but both classes are important: If the class distribution is highly skewed (such as 80:20 or 90:10), then a ...


25

The "baseline curve" in a PR curve plot is a horizontal line with height equal to the number of positive examples $P$ over the total number of training data $N$, ie. the proportion of positive examples in our data ($\frac{P}{N}$). OK, why is this the case though? Let's assume we have a "junk classifier" $C_J$. $C_J$ returns a random probability $p_i$ to ...


24

There is a lot of misunderstanding about evaluation. Part of this comes from the Machine Learning approach of trying to optimize algorithms on datasets, with no real interest in the data. In a medical context, it's about the real world outcomes - how many people you save from dying, for example. In a medical context Sensitivity (TPR) is used to see how ...


24

The Dice coefficient (also known as Dice similarity index) is the same as the F1 score, but it's not the same as accuracy. The main difference might be the fact that accuracy takes into account true negatives while Dice coefficient and many other measures just handle true negatives as uninteresting defaults (see The Basics of Classifier Evaluation, Part 1). ...


22

For multi-label classification you have two ways to go First consider the following. $n$ is the number of examples. $Y_i$ is the ground truth label assignment of the $i^{th}$ example.. $x_i$ is the $i^{th}$ example. $h(x_i)$ is the predicted labels for the $i^{th}$ example. Example based The metrics are computed in a per datapoint manner. For each ...


22

Short answer is: YES. Average Precision is a single number used to summarise a Precision-Recall curve: You can approximate the integral (area under the curve) with: Please take a look at this link for a good explanation.


21

If you spell out the definitions of precision (aka positive predictive value PPV) and recall (aka sensitivity), you see that they relate to one class independent of any other classes: Recall or senstitivity is the proportion of cases correctly identified as belonging to class c among all cases that truly belong to class c. (Given we have a case truly ...


20

There is no magic cut-off for either AUC-ROC or AUC-PR. Higher is obviously better, but it is entirely application dependent. For example, if you could successfully identify profitable investments with an AUC of 0.8 or, for that matter anything distinguishable from chance, I would be very impressed and you would be very rich. On the other hand, classifying ...


20

These are not the same thing and they are often used in different contexts. The Dice score is often used to quantify the performance of image segmentation methods. There you annotate some ground truth region in your image and then make an automated algorithm to do it. You validate the algorithm by calculating the Dice score, which is a measure of how similar ...


19

F1 score is applicable for any particular point of the ROC curve. This point may represent for example a particular threshold value in a binary classifier and thus corresponds to a particular value of precision and recall. Remember, F score is a smart way to represent both recall and precision. For F score to be high, both precision and recall should be ...


19

Letting $\beta$ be the weight in the first definition you provide and $\tilde\beta$ the weight in the second, the two definitions are equivalent when you set $\tilde\beta = \beta^2$, so these two definitions represent only notational differences in the definition of the $F_\beta$ score. I have seen it defined both the first way (e.g. on the wikipedia page) ...


18

Here is some discuss of coursera forum thread about confusion matrix and multi-class precision/recall measurement. The basic idea is to compute all precision and recall of all the classes, then average them to get a single real number measurement. Confusion matrix make it easy to compute precision and recall of a class. Below is some basic explain about ...


18

It's not that $\text{Precision} + \text{Recall}$ is a bad measure per se, its just that, on its own, the resulting number doesn't represent anything meaningful. You are on the right track though... what we are looking for is a combined, average of the two performance measures since we don't want to have to choose between them. Recall that precision and ...


18

In general, harmonic means are preferred when one is trying to average rates, instead of whole numbers. In the case of an F1-measure, a harmonic mean will penalize very small precisions or recalls whereas the unweighted arithmetic mean won't. Imagine averaging 100% and 0%: Arithmetic mean is 50% and Harmonic mean is 0%. The harmonic mean requires that both ...


16

As of July 2016, the package PRROC works great for computing both ROC AUC and PR AUC. Assuming you already have a vector of probabilities (called probs) computed with your model and the true class labels are in your data frame as df$label (0 and 1) this code should work: install.packages("PRROC") require(PRROC) fg <- probs[df$label == 1] bg <- probs[...


16

I've found He and Garcia (2009) to be a helpful review of learning in imbalanced class problems. Here are a few definitely-not-comprehensive things to consider: Data-based approaches: One can undersample the majority class or oversample the minority class. (Breiman pointed out that this is formally the equivalent to assigning non-uniform misclassification ...


15

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion) The definitions ...


14

Using sklearn and numpy: from sklearn.metrics import confusion_matrix import numpy as np labels = ... predictions = ... cm = confusion_matrix(labels, predictions) recall = np.diag(cm) / np.sum(cm, axis = 1) precision = np.diag(cm) / np.sum(cm, axis = 0) To get overall measures of precision and recall, use then np.mean(recall) np.mean(precision)


14

The reason for defining the F-beta score with $\beta^{2}$ is exactly the quote you provide (i.e. wanting to attach $\beta$ times as much importance to recall as precision) given a particular definition for what it means to attach $\beta$ times as much importance to recall than precision. The particular way of defining the relative importance of the two ...


14

Besides the AUC and Kohonen's kappa already discussed in the other answers, I'd also like to add a few metrics I've found useful for imbalanced data. They are both related to precision and recall. Because by averaging these you get a metric weighing $TP$s and both types of errors ($FP$ and $FN$): F1 score, which is the harmonic mean of precision and recall. ...


13

Almost all of scikit-learn's classifiers can give decision values (via decision_function or predict_proba). Based on the decision values it is straightforward to compute precision-recall and/or ROC curves. scikit-learn provides those functions in its metrics submodule. A minimal example, assuming you have data and labels with appropriate content: import ...


13

Yes, your assumptions about Kappa seem about right. Kappa as single, scalar metrics is mostly and advantage over other single, scalar metrics like accuracy, which will not reflect prediction performance of smaller classes (shadowed by performance of any much bigger class). Kappa solves this problem more elegantly, as you pointed out. Using a metric like ...


13

First, the claim on the Kaggle post is bogus. The paper they reference, "The Relationship Between Precision-Recall and ROC Curves", never claims that PR AUC is better than ROC AUC. They simply compare their properties, without judging their value. ROC curves can sometimes be misleading in some very imbalanced applications. A ROC curve can still look pretty ...


Only top voted, non community-wiki answers of a minimum length are eligible