Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
169
votes
3answers
70k views

ROC vs precision-and-recall curves

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other. Do they always provide complementary insight about the performance of a ...
76
votes
8answers
89k views

How to compute precision/recall for multiclass-multilabel classification?

I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have ...
94
votes
3answers
138k views

How do you calculate precision and recall for multiclass classification using confusion matrix?

I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I ...
15
votes
2answers
7k views

What is “baseline” in precision recall curve

I'm trying to understand precision recall curve, I understand what precision and recall are but the thing I don't understand is the "baseline" value. I was reading this link https://classeval....
16
votes
3answers
13k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
15
votes
3answers
9k views

Suggestions for cost-sensitive learning in a highly imbalanced setting

I have a dataset with a few million rows and ~100 columns. I would like to detect about 1% of the examples in the dataset, which belong to a common class. I have a minimum precision constraint, but ...
23
votes
3answers
24k views

Classification/evaluation metrics for highly imbalanced data

I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations. http://blog.revolutionanalytics.com/2016/...
3
votes
1answer
1k views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...
30
votes
4answers
12k views

Optimising for Precision-Recall curves under class imbalance

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, ...
21
votes
4answers
17k views

What are correct values for precision and recall in edge cases?

Precision is defined as: p = true positives / (true positives + false positives) Is it correct that, as true positives and <...
15
votes
3answers
6k views

ROC vs Precision-recall curves on imbalanced dataset

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset. For example, we have 10 samples in test dataset. 9 samples are positive and 1 is ...
12
votes
1answer
10k views

How to form a Precision-Recall curve when I only have one value for P-R?

I have a data mining assignment where I make a content-based image retrieval system. I have 20 images of 5 animals. So in total 100 images. My system returns the 10 most relevant images to an input ...
7
votes
1answer
1k views

Is it correct to use Precision-Recall AUC in a balanced dataset situation?

I have a binary classification scenario with a dataset that is unbalanced (much more negatives than positives). When I train a classifier on this dataset I get a Precision-Recall AUC of 0.7. Then I ...
5
votes
1answer
1k views

Confusion matrix, metrics, & joint vs. conditional probabilities

In the binary classification/prediction problem we have unknown labels $y\in\{0,1\}$, which we try to predict using an estimator $\hat{y}$. Commonly the performance of an estimator is summarized using ...
2
votes
1answer
2k views

Calculate precision and recall

I am really confused about how to calculate precision and recall in supervised machine learning algorithm using NB classifier with more than two classes. Say for example I have three classes $A$, $...
3
votes
2answers
1k views

Better in ROC AUC vs. better in PR AUC [duplicate]

I'm comparing two classification models by computing the area under ROC and Precision-Recall curves. However sometimes one model is better with AU-ROC but worse in AU-PR, and other times it's better ...
45
votes
9answers
112k views

How to interpret F-measure values?

I would like to know how to interpret a difference of f-measure values. I know that f-measure is a balanced mean between precision and recall, but I am asking about the practical meaning of a ...
24
votes
1answer
18k views

F1/Dice-Score vs IoU

I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar ...
15
votes
2answers
28k views

Increasing number of features results in accuracy drop but prec/recall increase

I am new to Machine Learning. At the moment I am using a Naive Bayes (NB) classifier to classify small texts in 3 classes as positive, negative or neutral, using NLTK and python. After conducting ...
32
votes
1answer
55k views

what does the numbers in the classification report of sklearn mean?

I have below an example I pulled from sklearn 's sklearn.metrics.classification_report documentation. What I don't understand is why there are f1-score, precision and recall values for each class ...
29
votes
2answers
32k views

Area under Precision-Recall Curve (AUC of PR-curve) and Average Precision (AP)

Is Average Precision (AP) the Area under Precision-Recall Curve (AUC of PR-curve) ? EDIT: here is some comment about difference in PR AUC and AP. The AUC is obtained by trapezoidal interpolation ...
16
votes
5answers
28k views

What impact does increasing the training data have on the overall system accuracy?

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-...
14
votes
3answers
15k views

What are the differences between AUC and F1-score?

F1-score is the harmonic mean of precision and recall. The y-axis of recall is true positive rate (which is also recall). So, sometime classifiers can have low recall but very high AUC, what that ...
13
votes
3answers
24k views

What are correct values for precision and recall when the denominators equal 0?

Precision is defined as: p = true positives / (true positives + false positives) What is the value of precision if (true positives + false positives) = 0? Is it just undefined? Same question for ...
11
votes
5answers
28k views

How to calculate precision and recall in a 3 x 3 confusion matrix

...
10
votes
1answer
919 views

Why doesn't recall take into account true negatives?

Why doesn't recall take into account true negatives? In experiments where true negatives are just as important as true positives, is their a comparable metric that does take it into account?
8
votes
4answers
19k views

Average Precision in Object Detection

I'm quite confused as to how I can calculate the AP or mAP values as there seem to be quite a few different methods. I specifically want to get the AP/mAP values for object detection. All I know for ...
11
votes
3answers
6k views

Classifier with adjustable precision vs recall

I am working on a binary classification problem where it is much more important to not have false positives; quite a lot of false negatives is ok. I have used a bunch of classifiers in sklearn for ...
8
votes
2answers
4k views

Does it make sense to measure recall in recommender systems?

Assume I've built a recommender system that (given say movie rankings or whatever of many users) will produce a list of 10 recommended movies for each user to watch. Imagine that I also have some ...
6
votes
1answer
3k views

Why the sum of true positive and false positive does not have to be equal to one?

While reading this question above, I got confused about the sum of true positive and false positive. If an aircraft is present in a certain area, a radar detects it and generates an alarm signal ...
8
votes
3answers
17k views

Precision and recall for clustering?

Im confused about calculating precision and recall for clustering mentioned in this paper, Model-based Overlapping Clustering, A Banerjee et al., (last paragraph of column 1 on page 5). Suppose, if ...
7
votes
1answer
2k views

Comparable training and test cross-entropies result in very different accuracies

Premises I'm training a convolutional neural network (ConvNet) on 51 subclasses in the ImageNet dataset. In order to keep an eye on overfitting, I have been suggested to plot training and testing ...
4
votes
2answers
15k views

Threshold in precision/recall curve

While I was reading Torgo's Data Mining with R, I found that the description of precision/recall curve was different compared with other approaches. Usually, these curves are based on a threshold that ...
4
votes
1answer
10k views

How to improve F1 score with skewed classes?

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1. I've tried several classification algorithms, without too much fine tuning, just to get a ...
7
votes
1answer
4k views

Evaluating recommender systems with (implicit) binary ratings only

I'm analyzing a set of news articles and user libraries. User library is the set of news articles shared by one user. Obviously, the rating is 1 (the article is in user's library) and 0, otherwise. I ...
7
votes
1answer
7k views

What is AUC of PR-curve?

I understand that AUC under ROC curve is a classic evaluation measurement for classifiers (which is basically the accuracy). However, when data is imbalanced, PR will be alternative. So, what does the ...
7
votes
4answers
2k views

Evaluating and combining methods based on ROC and PR curves

I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am ...
6
votes
1answer
444 views

Why AUC-PR increases when the number of positives increase?

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the ...
8
votes
1answer
2k views

Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to ...
5
votes
2answers
6k views

What does it imply if accuracy and recall are the same?

I did a number of machine learning experiments to predict a binary classification. I measured precision, recall and accuracy. I noticed that my precision is generally quite high, and recall and ...
2
votes
1answer
337 views

Area Under the Precision Recall curve -similar interpretation to AUROC?

I am trying to interpret the AUCPR. Say I have the following Precision-Recall curve. Firstly: It ends at 0.38 on the y-axis because this particular plot has ...
0
votes
0answers
1k views

2 Different F1-Measure to calculate clustering performance - which one is correct and why?

I know it sounds incorrect but that is the truth Here let me show you This below one is the first one and very widely used in the literature First one reference : Steinbach, Michael, George Karypis,...
0
votes
0answers
19 views

What is the standard metric used in recommendation systems to evaluate the rankings?

I was searching for a metric to do this for a while and still could not find. More specifically, my problem is as follows. I have a ranked golden corpus. For example, consider that it looks as ...
11
votes
3answers
4k views

How to choose a good operation point from precision recall curves?

Is there any standard method to determine an "optimal" operation point on a precision recall curve? (i.e., determining the point on the curve that offers a good trade-off between precision and recall) ...
4
votes
1answer
688 views

How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics?

How can Precision-Recall (PR) curves be used to judge overall classifier performance when Precision and Recall are class based metrics? Since in a binary classifier, there are two classes, often ...
3
votes
2answers
739 views

What is the effect of training a model on an imbalanced dataset & using it on a balanced dataset?

When evaluating a model, for example a binary classifier, should the train and test set have 50% + and 50% - label distribution or could the distribution be random? If the distribution is biased in ...
2
votes
1answer
3k views

Can a Precision-Recall curve or a ROC curve be horizontal?

I am working on a binary classification task on imbalanced data. Since the accuracy is not so meaningful in this case. I use Scikit-Learn to compute the Precision-Recall curve and ROC curve in order ...
0
votes
0answers
125 views

ROC curve interpretation [duplicate]

In the context of binary classification how do you interpret ROC curve: more precisely: 1) Why the diagonal stand for a random classifier? [Edit] Let's imagine a random classifier: each time he ...