Questions tagged [precision-recall]
P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.
496
questions
1
vote
1
answer
28
views
+50
Choosing the correct evaluation metric between F1-score and Area under the Precision-Recall Curve (AUPRC)
We're currently working on detecting specific objects (e.g. poultry farms, hospitals) from satellite images. We've modeled the problem as a binary image classification task (i.e. classifying images ...
- 729
0
votes
0
answers
14
views
Is it possible to estimate the number of positives from precision and recall values?
Let's say, I have a binary predictor, and its performance in precision and recall is known from the previous study. Now, we apply the predictor on the new (unknown) dataset with 1000 samples, and got ...
- 103
0
votes
1
answer
664
views
Multiclassification: precision-recall from scratch vs sklearn
I would like to know if there´s any issue behind using sklearn's precision/recall metric functions and coding up from scratch in ...
CommunityBot
- 1
0
votes
1
answer
244
views
Confidence interval for Accuracy, Precision and Recall
Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that ...
CommunityBot
- 1
261
votes
7
answers
125k
views
ROC vs precision-and-recall curves
I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.
Do they always provide complementary insight about the performance of a ...
- 101
0
votes
2
answers
165
views
Are there any difference using scores or probabilities for roc_auc_score and precision_recall_curve functions?
I'm working with a GNN model for link prediction and using precision_recall_curve and roc_auc_score from the ...
- 159
80
votes
4
answers
89k
views
F1/Dice-Score vs IoU
I was confused about the differences between the F1 score, Dice score and IoU (intersection over union). By now I found out that F1 and Dice mean the same thing (right?) and IoU has a very similar ...
13
votes
2
answers
10k
views
Which performance metrics for highly imbalanced multiclass dataset?
I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly ...
- 111
4
votes
2
answers
360
views
Relationship between recall and Precision-Recall curve
I'm trying to evaluate some classification algorithms' results in my imbalanced dataset. By imbalanced, I mean there are much more negative than positive labels. Accuracy and precision are always good,...
CommunityBot
- 1
0
votes
0
answers
27
views
How to calculate AUC for a P-R curve with unusual starting point
I am working with a binary classifier that is outputting scores between 0 and 1, indicating probabilities of class membership, according to the model. I produced a P-R curve and the first point (i.e., ...
- 78.1k
2
votes
2
answers
365
views
Estimating "prevalence" from a classifier's precision and recall?
I am doing an information retrieval task where the goal is to estimate the total number of positive documents as opposed to which document is positive or negative.
My approach has been focusing on ...
CommunityBot
- 1
1
vote
1
answer
480
views
Evaluation metric for time-series anomaly detection
My dataset is time-series sensor data and anomaly ratio is between 5% and 6%
1.
For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ?
When empirically ...
CommunityBot
- 1
0
votes
1
answer
560
views
High Precision and low recall score for TF-IDF when using KNN algorithm
I have twitter data which is labelled with the sentiment(Postive, Negative, Neutral) and I have evaluated the performance of Tf-Idf and Doc2Vec feature extractor using the KNN algorithm and logistic ...
CommunityBot
- 1
0
votes
1
answer
431
views
Why macro F1 measure can't be calculated from macro precision and recall?
I'm interested in calculating macro f1-score by macro precision and recall manually. But the results aren't equal. What is the difference in the final formula between f1 and f1_new in code?
Would you ...
CommunityBot
- 1
1
vote
1
answer
260
views
Calculating the confidence interval or standard error of a PPV adjusted for prevalence
I am trying to evaluate how well a disease test performs in a case-control study. In this example, the prevalence is 0.5%, and the results are below:
Disease +
Disease -
Test +
40 (TP)
10 (FP)
Test ...
CommunityBot
- 1
3
votes
2
answers
263
views
Is F-score the same as accuracy when there are only two classes of equal size?
The title says it all: Is F-score the same as accuracy when there are only two classes of equal sizes?
For my specific case, I have measurements of a group of people under two different situations and ...
- 9,327
2
votes
1
answer
2k
views
Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?
On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
CommunityBot
- 1
2
votes
1
answer
36
views
What is the best method to calculate confidence intervals on precision and recall which are not independent?
I have a classification model which classifies user's bank transactions into two categories. From this model I produce precision and recall metrics.
I would like to understand the confidence around ...
- 1,246
3
votes
2
answers
308
views
How to calculate p-value to compare AUPRC for two models?
I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
CommunityBot
- 1
1
vote
1
answer
55
views
How to define Precision when we have multiple predictions for each ground truth instance?
In my problem, it is possible to have multiple predictions for a ground truth instance. How we define precision in such scenarios?
For further clarification consider the following example. We have 1 ...
- 607
1
vote
0
answers
22
views
Precision and Recall of a combined classifier
I have two classifiers, trained on the same dataset, each predicting a different variable. let's call these X1 and X2. They have their respective precision and recall measures X1-p,r, and X2-p,r.
I ...
- 62.4k
3
votes
2
answers
97
views
Calculating "accuracy", "recall" etc. without classification
I have a set of models, that I'm comparing to each other with respect to prediction of a binary event. I'm using a few proper scores (Brier, log), but I also need accuracy, recall, sensitivity etc., ...
- 1,127
1
vote
1
answer
901
views
Logistic Regression: What is the value for precision when recall (true positive rate) is 0?
A quick overview of definitions before I get into the question:
True Positive (TP): An actual positive that the model classified as positive
False Positive (FP): An actual negative that the model ...
CommunityBot
- 1
0
votes
0
answers
14
views
Why would area under the PR curve include points off of the Pareto front?
(Let's set aside thoughts about if we should be calculating PR curves or areas under them at all.)
A precision-recall curve for a "classification" model can contain points that should not be ...
- 62.4k
2
votes
1
answer
75
views
Comparing probability threshold graphs for F1 score for different models
Below are two plots, side-by side, for an imbalanced dataset.
We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
- 4,524
2
votes
2
answers
405
views
Adjusting precision recall curve for oversampling
I built a model for a binary target using oversampled data.
The population target prevalence is 0.25.
I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the ...
CommunityBot
- 1
1
vote
1
answer
364
views
Interpreting precision and recall graphs
i have a CNN for sentiment analysis whose precision and recall for validation data over 10 training epochs is (average:macro):
The dataset contains more positive samples than negatives.I have ...
CommunityBot
- 1
1
vote
1
answer
2k
views
How to interpret precision and recall for multiclass prediction?
I have a few models doing prediction with 4 classes, with the output precision and recall varying with different labels.
For example I have (with the class labels being 0, 1, 2, 3 on the x axis):
I ...
CommunityBot
- 1
1
vote
1
answer
981
views
High Precision and High Recall issue- Random Forest Classification
I am building a classification model using Random Forest technique using GridSearchCV. The target variable is binary where 1 is 7.5% of total population. I have used several values of GridSearch ...
CommunityBot
- 1
3
votes
1
answer
382
views
Mean Average Precision (MAP) object detection metric
I am reading up on object detection metrics from this github repository. I have a slight confusion regarding the precision x recall curve mentioned under the Metrics heading. It says that
The ...
CommunityBot
- 1
8
votes
1
answer
603
views
Lack of rigor when describing prediction metrics
I constantly see metrics that measure the quality of a classifier's predictions, such as TPR, FPR, Precision, etc., being described as probabilities (see Wikipedia, for example).
As far as I know, ...
- 62.4k
2
votes
3
answers
1k
views
Precision and recall in content-based recommender
I have some trouble understanding the concept of using precision and recall to evaluate a content-based recommender.
Suppose I want to recommend articles to users. A content-based recommender will ...
CommunityBot
- 1
31
votes
5
answers
85k
views
What impact does increasing the training data have on the overall system accuracy?
Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system?
When do we detect that adding more training data could possibly over-...
- 1,148
0
votes
1
answer
630
views
How to measure classifier performance on small and skewed sample dataset?
I have a small sample dataset (n=25) that represents the ground truth for a larger set (n=10k). I am doing a classification task and obtain, say, 3 true positives, 20 true negatives, 1 false positive, ...
CommunityBot
- 1
0
votes
0
answers
20
views
Can the Log of PR AUC curve plot be any useful?
I was doing some tests regarding my PR curve for 2 different models (first image), and I got the idea of ploting the log of those curves (second image) to see if there were any insights that I could ...
0
votes
0
answers
15
views
How do you calculate multiclass microaverage precision and recall?
I have a confusion matrix that is 4 class.
...
1
vote
0
answers
44
views
Confidence intervals for Object Detection metrics
I would like to come back on this "When do we require to calculate the confidence Interval?" since recently a reviewer asked me to provide confidence intervals for metrics regarding my work ...
- 44.2k
1
vote
1
answer
94
views
Why does my PR Curve look like this?
These are my recall and precision stats for the model I built. The Curve does not look good where recall is 0. Not sure why there are so many points there. Can anyone help and explain why the curve ...
- 3,674
3
votes
3
answers
137
views
Isn't partial AUC a better metric than AUC for cost-sensitive classification problems?
In many classification problems, the cost of a FP is different from the cost of a FN. In spam detection, a FP (a regular email classified as spam) should have a high cost. In cancer prediction, a FN (...
- 361
3
votes
2
answers
141
views
Calculate area under precision-recall curve from area under ROC curve and the prevalence
I am reading material that reports the area under a ROC curve. I am curious to know what the performance would be in precision-recall space. From the sensitivity and specificity values in the ROC ...
- 1,023
1
vote
1
answer
77
views
Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?
It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
- 127
0
votes
0
answers
75
views
Is Hyperparameter Tuning for Maximized Recall a Bad Thing?
I have a somewhat theoretical question: I work in an area that requires a number of anomaly detection solutions. When we approach these problems, we cross-validate and for each fold, we oversample ...
- 51
3
votes
1
answer
238
views
Can precision and recall of a DNN trained on human-labeled data be higher than precision and recall of the humans who labeled the data?
I was discussing Deep Learning with an academic statistician, who was criticizing the field as "lacking scientific rigor, overhyped and delivering results which are way worse than claimed". In ...
- 1,420
0
votes
0
answers
27
views
F1 score for multiple datasets
I am testing my algorithm on 20 different datasets. So, as a result I derived 20 F1 scores based on precision and recall for each run (dataset). Is there any way to combine all these 20 F1 scores into ...
0
votes
0
answers
19
views
How should the prevalence be calculated when determining prevalence-adjusted PPV?
I am assessing the performance of a test for a certain disease. I have selected the disease cases and controls from a large dataset that is reasonably representative of the general population.
As I ...
- 11
2
votes
1
answer
106
views
Poor balanced accuracy and minority recall but perfect calibration of probabilities? Imbalanced dataset
I have a dataset with a class imbalance in favour of the positive class (85% occurence)
I'm getting a fantastically calibrated probabilities profile but balanced accuracy is 0.65 and minority recall ...
- 62.4k
1
vote
0
answers
27
views
How to choose k for MAP@K?
Scenario: We want to evaluate our recommender system, which recommends items to potential customers when visiting a product detail page.
Here are actual relevant items:
...
- 977
0
votes
1
answer
874
views
Why does sklearn list weighted precision and micro precision separately if they are the same thing?
This post explains that micro precision is the same as weighted precision.
(And the logic applies to recall and f-score as well.)
So why does sklearn.metrics list micro and weighted as separate ...
CommunityBot
- 1
1
vote
2
answers
1k
views
P-value for precision recall curve significance
I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a ...
- 101
2
votes
1
answer
307
views
Sudden drop to zero for precision recall curve
I am training a neural network classifier with 250k training samples and 54k validation samples. The output activation is sigmoid. I noticed a sudden drop in the precision for the very top probability ...
- 44.2k