Questions tagged [precision-recall]
P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.
482
questions
0
votes
0
answers
17
views
How to choose k for MAP@K?
Scenario: We want to evaluate our recommender system, which recommends items to potential customers when visiting a product detail page.
Here are actual relevant items:
...
- 887
0
votes
0
answers
11
views
Aggregating predictions on micro data
I am a machine-learning noob, so please bear with me.
I am trying to predict the aggregate number of businesses that will exit (i.e. shut down permanently) in the next quarter (or year). However, my ...
2
votes
1
answer
38
views
Sudden drop to zero for precision recall curve
I am training a neural network classifier with 250k training samples and 54k validation samples. The output activation is sigmoid. I noticed a sudden drop in the precision for the very top probability ...
- 21
0
votes
0
answers
13
views
Confidence intervals for Object Detection metrics
I would like to come back on this "When do we require to calculate the confidence Interval?" since recently a reviewer asked me to provide confidence intervals for metrics regarding my work ...
- 101
1
vote
1
answer
61
views
Why use average_precision_score from sklearn? [duplicate]
I have precision and recall values and want to measure an estimator performance:
...
- 31
0
votes
0
answers
14
views
Methods to approximate Area under Precision-Recall Curve
average_precision_score from sklearn uses formula:
ap = sum( (recall[k+1] - recall[k]) * precision[k+1] )
But trapezoidal rule implies:
...
- 31
0
votes
0
answers
35
views
How to calculate Mean Average Precision at a fixed IoU threshold?
Papers such as COCO and the VisDrone often list MAP, MAP50, MAP75 in their evaluation criteria. To my understanding, MAP is the averaged AP for all classes across an IoU range (0.5 - 0.95 for example),...
0
votes
0
answers
13
views
Recall and precision point estimates (statistical inference)
Let's say I have a population of 1M objects. I want to make a binary classifier to use on that data, but I can't manually classify all the 1M to create training data because that would take too long, ...
5
votes
2
answers
417
views
Understanding Precision Recall in business context
So, I know yet another Precision, Recall question which is asked umpteenth times now.
I wanted to ask some specific business related questions.
Imagine if you are building a classifier to predict ...
- 1,075
2
votes
1
answer
27
views
Binary classification metrics - Combining sensitivity and specificity?
The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN).
For ...
- 1,157
1
vote
0
answers
17
views
How to start GNN optimization to get higher precision?
I'm developing a GNN for missing links prediction following this blog post for PyG library. I'm using almost the same GNN with a different dataset. Altough my dataset is similar to the MovieLens ...
- 75
0
votes
1
answer
58
views
Are there any difference using scores or probabilities for roc_auc_score and precision_recall_curve functions?
I'm working with a GNN model for link prediction and using precision_recall_curve and roc_auc_score from the ...
- 75
2
votes
1
answer
29
views
Precision and recall reported in classification model
I have one question about the evaluation metrics of classification models. I see many people report the precision and recall value for their classification models. Do they choose a threshold to ...
- 459
1
vote
0
answers
96
views
Interpretation of area under the precision-recall curve
The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated.
This post gives the area under the precision-recall curve as ...
- 55.1k
2
votes
0
answers
31
views
Is there a way to effect the shape of precision-recall curve?
As long as I know, for both ROC and PR curves, the classifier performance is usually measured by the AUC. This might indicate that classifiers with equivalent performance might have different ROC/PR ...
- 360
1
vote
1
answer
44
views
combine specificity and
I am performing classification on an imbalanced dataset (70% negatives).
If a prediction is negative I take a specific action otherwise an opposite one. As in both cases some costs are implied, I want ...
- 285
1
vote
2
answers
67
views
Why don't we use the harmonic mean of sensitivity and specificity?
There is this question on the F-1 score, asking why we compute the harmonic mean of precision and recall rather than its arithmetic mean. There were good arguments in the answers in favor of the ...
- 211
3
votes
1
answer
225
views
When to use a ROC Curve vs. a Precision Recall Curve?
Looking for the circumstances of when we should use a ROC curve vs. a Precision Recall curve.
Example of answers I am looking for:
Use a ROC Curve when:
you have a balanced or imbalanced dataset (...
- 704
3
votes
1
answer
187
views
ROC AUC has $0.5$ as random performance. Does PR AUC have a similar notion?
In considering ROC AUC, there is a sense in which $0.5$ is the performance of a random model. Conveniently, this is true, no matter the data or the prior probability of class membership; the ROC AUC ...
- 55.1k
11
votes
8
answers
6k
views
My machine learning model has precision of 30%. Can this model be useful?
I've encountered an interesting discussion at work on interpretation of precision (confusion matrix) within a machine learning model. The interpretation of precision is where there is a difference of ...
- 121
0
votes
0
answers
37
views
Correctly evaluating unsupervised learning model
I am trying to compare various unsupervised machine learning models to detect anomalous water consumption in each user's house. Now I have 10 datasets (minutely data, no anomalous points) that have no ...
- 101
0
votes
0
answers
15
views
Flipping inputs in multilabel classification
I have framed a classification problem as follows:
I have $N$ items, and wish to predict a set of relevant tags for each out of $M$ tags. An item can have anywhere from 0 to $M$ applicable tags.
To ...
- 1
0
votes
0
answers
113
views
Confidence interval for Accuracy, Precision and Recall
Classification accuracy or classification error is a proportion or a ratio. It describes the proportion of correct or incorrect predictions made by the model. Each prediction is a binary decision that ...
- 277
1
vote
0
answers
78
views
A better linear model has less precision(relative to the worse model) at a larger threshold
I trained two models using the same algorithm - logistic regression (LogisticRegression(max_iter=180, C=1.05) for ~27 features and ~330K observations). I used the ...
0
votes
0
answers
69
views
Is weighting still needed when using undersampling?
I have an model that I want to test with my existing data to calculate precision, recall etc.
The data is actually unbalanced dataset: Class A 70%/Class B 30%.
I created a data set by undersampling ...
1
vote
0
answers
103
views
Accounting for overrepresentation of positives in binary classification test set for calculation of precision and recall
I have a binary classification task with highly imbalanced data, since the class to be detected (in the following referred to as the positives) is very rare.
For data limitation reasons my test set ...
- 11
0
votes
0
answers
96
views
Raising decision threshold reduces classifier precision
I'm using neural network to detect specific patterns in a video. Currently I use RCNN-like approach, where first network finds candidate regions and second one classifies small video fragments into ...
- 101
0
votes
0
answers
15
views
Weighted precision & recall - With class weights vs oversampling
If weighted precision is calculated as
...
- 101
0
votes
0
answers
155
views
Comparing AUC-PR between groups with different baselines
So I know that the area under the precision-recall curve is often a more useful metric than AUROC when dealing with highly imbalanced datasets. However, while AUROC can easily be used to compare ...
- 2,126
0
votes
0
answers
38
views
Does the Precision-Recall AUC approach the ROC AUC as the data becomes balanced?
I am working on a Machine Learning classifier. It is a binary response and most predictor variables are categorical. I have several years of data and for some years, the response is imbalanced (more ...
- 151
1
vote
0
answers
54
views
F1 Score vs PR Curve
If I understood correctly, PR Curve it's just the mean of F1 score computed multiple times with different thresholds.
In the task of outlier detection those are two suggested metrics given the fact ...
- 23
1
vote
1
answer
98
views
Precision-Recall straight line interpretation
In the precision-recall curve shown, how should I interpret the straight line from 0.0 to almost 6.0?
Update: I include the confusion matrix where the precision and recall come from.
1
vote
1
answer
145
views
Estimate recall for extremely rare events
I want to estimate the recall of a binary classifier. I have a dataset of ~1B examples but I don't know the ground truth, the only thing I know is that positives are extremely rare. I can randomly ...
- 13
3
votes
1
answer
126
views
Singular beta in the F-beta vs. threshold score?
Consider this plot of the $F_\beta$ score for different values of $\beta$. I have a hard time getting an intuition as to why they intersect at a same point. (Cf. this blog post.) In other words, why ...
- 755
2
votes
2
answers
742
views
How to get the threshold from PrecisionRecallDisplay?
My goal is to tune the Classifier with probability predict_proba() < threshold. Therefore, I need to get the threshold.
The problem is ...
1
vote
0
answers
38
views
Optimal metric for training with Class-specific masked input features and imbalanced dataset
I have a classification problem of 8-classes, which are extremely imbalanced. The input dataset consists of sequences, each of length n features, where n = 19. For each of the 8 classes, I have a ...
- 111
1
vote
0
answers
159
views
Calculating the confidence interval or standard error of a PPV adjusted for prevalence
I am trying to evaluate how well a disease test performs in a case-control study. In this example, the prevalence is 0.5%, and the results are below:
Disease +
Disease -
Test +
40 (TP)
10 (FP)
Test ...
- 11
2
votes
2
answers
83
views
Isn't partial AUC a better metric than AUC for cost-sensitive classification problems?
In many classification problems, the cost of a FP is different from the cost of a FN. In spam detection, a FP (a regular email classified as spam) should have a high cost. In cancer prediction, a FN (...
- 1,157
1
vote
0
answers
14
views
Can someone explain why finding similar embeddings coming from two different net gives bad recall?
I'm currently working on an ensemble of 5 differently trained networks using MinkLoc3D v2 as base-net.
I'm currently investigating the reason for lousy recall when I compare the extracted database ...
- 11
2
votes
2
answers
229
views
Sensitivity vs. specificity vs. recall
Given a binary confusion matrix with true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), what are the formulas for sensitivity, specificity, and recall?
I'm ...
- 219
0
votes
0
answers
56
views
Precision-Recall curve in terms of conditional probability
To plot the PR-curve, we need to construct precision in terms of bayes theorem
$$
Precision = P(Y=1|Y'=1) = P(Y=1| q>q_t)
$$
where $Y$ is the true class, $Y'$ predicted class and $q_t$ is a ...
- 101
0
votes
0
answers
792
views
Plotting precision-recall curve using plot_precision_recall_curve and precision_recall_curve results in different plots
I am plotting the precision-recall curves for my models which I have built using an imbalanced dataset.
I initially plotted the precision-recall curve for my models using the ...
- 133
0
votes
0
answers
159
views
Choosing 'scoring' parameter in random search
i'm still confused about scoring parameter in randomized search
First i want to know if my machine learning model is overfit or not. I use roc auc score between train and test.. turns out there is ...
0
votes
0
answers
132
views
What Does AUC for Precision Recall Curve Stand For?
Similarly to What does AUC stand for and what is it?, I'd like to know the interpretation of the AUC for the Precision Recall curve.
One can calculate the precision recall curve:
Then easily ...
- 395
1
vote
1
answer
38
views
High recall or precision for information retrieval system?
For disease detection, we need a high recall.
For information retrieval system, do we need a high precision?
- 315
3
votes
1
answer
51
views
Precision Recall Curve Intepretation
I have made the Precision-Recall for my model. The red line is the prevalence. I do not understand the fluctuations in the beginning. Should it be more smooth?
- 139
1
vote
0
answers
18
views
Does a model with 0.5 AUROC imply an average precision equal to the proportion of positive examples?
A random model has an area under the ROC curve equal to 0.5.
We also know that a random model has an area under the Precision-Recall curve equal to the proportion (p) of positive examples.
Then, here'...
- 111
0
votes
0
answers
21
views
How good do accuracy, precision, recall, and F1 have to be to use for flight critical environments?
How good do accuracy, precision, recall, and F1 have to be to use for flight critical environments?
Maybe there are rules of thumb, general practices, best practices, lessons learned, etc.
Flight ...
- 101
0
votes
1
answer
139
views
How can area under ROC (AUC) be bad when precision, recall, and accuracy are all good?
I have a model with the following scores:
Precision: 0.703588
Recall: 0.976526
Accuracy: 0.694936
I thought this was fairly decent, especially considering that my (binary) response class is 1/3 of the ...
- 247
1
vote
1
answer
57
views
Reporting performance measures for classification in percentage or fraction?
I have seen classification metrics like f1-score, precision and recall being reported both as fractions and percentages. These measures are between 0 and ...
- 23