Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Perfect recall but moderate precision due to imbalance?

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended ...
1
vote
0answers
14 views

How to interpret the Precision and Recall curve in-sample vs out-of-sample

I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best ...
0
votes
0answers
8 views

Calculating classification metrics when “true” label is also generated by another classification model

I have a binary classification model $A$ and want to calculate its precision and recall for positive and negative classes. The "ground truth" or "true" labels for this model are ...
0
votes
2answers
40 views

Is it ok a threshold of 0?

I am dealing with a classification problem with a dataset containing 60k rows: 69k are negative class, and 1k is positive. I trained my models and I obtained the confusion matrices with a threshold of ...
0
votes
1answer
31 views

What value for recall implies a logistic regression model is good?

I'm studying logistic regression using Python and about metrics to have a good model, I know this three: accuracy, precision and recall. In the same way, I was studying using a dataset about ads in ...
0
votes
0answers
12 views

Dummy Classifier - No Skill Line on a Binary Classification Precision Recall Curve Using a Stratified Approach - How Calculated?

I am doing binary classification on a highly imbalanced data set where the positive value is about 1.6% of the dataset. I am training and running models but I want to compare against a dummy ...
0
votes
0answers
9 views

COCO evaluation - Negative values on AP and AR

I am trying to evaluate one of the models using COCO dataset metrics. For some of the precision metrics I am getting results that I can comprehend. For example:Average Precision (AP) @[ IoU=0.50:0.95 ...
0
votes
0answers
12 views

R: cannot find precision, recall and f-measured

I ran a logistic model, and want to know calculate precision, recall and f-measured from confusion matrix. Its accuracy is 0.995. ...
0
votes
1answer
22 views

Precision and Recall With A Binary Classifier

I want to calculate the precision and recall of my results, however, I'm a little confused on how to create the confusion matrix. I have a binary response either yes or no from my participants, ...
0
votes
0answers
6 views

Performance Metrics on Within Subjects Study

I ran a simple yes or no within-subject experiment. i.e there were 4 conditions, and each participant answered a single question for a subset of images per condition. My question is how would I ...
0
votes
1answer
12 views

What is the average precision in the case of no positives for a given category in the context of object detection

In attempting to calculate the average precision of an object detection model, I am wondering about an edge case. Suppose at evaluation time that for a given category, that no detections of that ...
0
votes
0answers
13 views

P-value for precision recall curve significance

I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a ...
0
votes
0answers
11 views

Interpretation of Precision and Recall from Object detection API?

In given picture I have precision and recall value -1.000. What does this signify? Could someone please help me to interpret this results? Furthermore Can I calculate F1 score for this average ...
0
votes
0answers
15 views

Is the PR AUC invariant under label flip?

The ROC-AUC curve is invariant under a flip of the labels. I don't know if it's a famous result, so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
0
votes
0answers
16 views

Does it make sense to mention F1 score on multiclassification problem?

I have two questions that came to my mind while reading some comments on a multiclassification problem: I read that models were evaluated using F1 score (which is a combination of Precision and ...
2
votes
2answers
132 views

Weighting common performance metrics by classification outcomes?

Cost-sensitive classification metrics are somewhat common (whereby correctly predicted items are weighted to 0 and misclassified outcomes are weighted according to their specific cost). Some examples ...
1
vote
0answers
29 views

Statistical test for comparing Precision-Recall curves

When evaluating models' performance using Receiver Operator Characteristic (ROC) curves, DeLong's test can be used to assess whether two curves are significantly different. Is there a statistical ...
0
votes
0answers
29 views

Can we use precision and recall in regression?

Precision is the ratio of true positive to all the positive results whereas recall is the ratio of true positives to all relevant results. As we know that precision and recall are classification ...
0
votes
1answer
12 views

How does recall help precision overcome “length-related problems?”

I'm reading the paper on the Bilingual Evaluation Understudy (BLEU) metric BLEU: A Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002) and had a question regarding a quote ...
0
votes
0answers
8 views

What is the right way to take a PR AUC average of a set of binary classifiers?

I have a multilabel classification problem. I am choosing to treat it as a set of independent binary classifiers. Each label has its corresponding skewness which can range from 1% to 20%. Suppose for ...
0
votes
0answers
17 views

I wonder why the precision of 1 class is much lower on the test set relative to the training set?

I used the LightGBM algorithm to predict the outcome of tennis matches. Next I made two confusion matrices, one for the training set and one for the test set. I calculated stats as precision, recall, ...
0
votes
0answers
12 views

High precision, but low recall on training data. What could be wrong?

I am training RNN for timeseries binary classification. I observed that network has high precision, but low recall on both training and testing data. I tried multiple architectures, but same problem ...
0
votes
0answers
5 views

Is it possible to identify which features contribute to the precision and recall for each label in sklearn's classification report output?

Below is an example of classification report output by sklearn. Is there an easy way to identify which features contribute to the precision and recall improvement for each label? I can start with ...
1
vote
0answers
38 views

Precision-Recall Curve and Area under Precision-Recall Curve (AUC)

I created model (logistic regression) and now trying to create Precision-Recall plot and calculate area under Precision-Recall Plot. I'd like to note that this model is defective: ...
0
votes
0answers
11 views

Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more ...
0
votes
0answers
22 views

Choosing between Average Precision and Precision for Classification Evaluation Metric

I am working on a classification problem in which I will be classifying between fraudulent and non fraudulent transactions. I will be detecting 100 transactions having the highest probability of ...
0
votes
0answers
22 views

Interpreting a precision recall curve

I am working on my ms thesis about predicting skin cancer I have these PR curves 1 - They start from 1.0 but they don't end at 1.0, is this a problem? 2 - SVM falls from 1.0 to 0.0 immediately and ...
1
vote
0answers
52 views

Why we use precision/recall in binary classification but sensitivity(=recall)/specificity in medicine?

Sensitivity=recall is used in both fields, but the second metric is different. Why? Both tasks (classification and medicine) look same - data has two classes and we do some predictions on it and want ...
0
votes
0answers
51 views

Precision-Recall Curve Intuition for Multi-Class Classification

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
0
votes
0answers
18 views

When to stop training a binary classifier when precision-recall curve and logits output is of interest

I have a binary classifier (DL model) and a balanced dataset (50-50 for the two classes, in both training and testing data). Let's say I want to optimize for precision. I can adjust my positive label ...
0
votes
0answers
78 views

how to find mean average precision of object detection algorithms

To start with, I would like to mention another question which was asked in a better way. But my problem differs. pseudo code for the algorithms I have four different object detection algorithms which ...
1
vote
0answers
69 views

Can precision and recall of both classes in a binary classification be compared?

While I was going through the question How to interpret classification report of scikit-learn? with the following metrics - I saw following claim (in the accepted answer) - you cannot compare the ...
3
votes
2answers
111 views

Disadvantage of precision at k

Suppose 10 documents were retrieved (rectangle with black color is relevant document). In the following table, Precision @ k is calculated. P@10 or "Precision at 10" corresponds to the ...
0
votes
0answers
24 views

Imbalanced classification what is good F1 score?

For imbalanced classification (say 85:15), what is good value of F1 score? An answer https://stats.stackexchange.com/a/217343/285091 says "Experiments indicate that the sweet spot here is around ...
0
votes
0answers
8 views

Model performance: accuracy equals recall for all experiments

I am working on a multiclass imbalanced dataset set. I would like to evaluate the model performance, but I cannot understand why the model accuracy is equal to the recall for all experiments. Class ...
1
vote
1answer
86 views

Starting point of the PR-curve and the AUCPR value for an ideal classifier

I have two questions about the PR-curve: What is the starting point of the PR-curve ? I mean the point which corresponds to the highest possible threshold (i.e. when all scores are below this ...
0
votes
1answer
25 views

imbalanced learning: precision vs recall trade-off

Working on a multi-class problem (five classes) for which the dataset is highly imbalanced (two classes with less than 2% samples). Which metric between precision ...
5
votes
1answer
27 views

Why to downweight Precision in nominator of F-beta when I actually want to upweight Precision?

F-beta score's formula calculates like this: $$ F_{\beta} = (1+ \beta^2) \frac{PR}{\beta^2P + R} $$ However, according to some sources, in case I want to add more emphasis to Precision I should use ...
0
votes
1answer
39 views

False positive rate at K recall

I just stumbled upon a new metric I've never heard about called False Positive rate at K recall (FPR-K). Searching the internet I just managed to find more papers using the metric but none of them ...
2
votes
2answers
152 views

Classifier can predict time series 1 day in advance, but not more. Why?

To ask the question more precisely: when doing Time Series classification, I observe the classifier prediction is good if test data directly follows (in chronology) the train data. But when the train ...
1
vote
0answers
51 views

Possible reasons that validation recall is fluctuating across different epochs but the precision is stable?

I know this is not a coding question, but didn't have any idea where can I ask for help on this. I'm training a deep learning model. After each epoch I measure the performance of the model on ...
1
vote
0answers
21 views

ROC Curve for data sets with large negative bias

For context, I've read this forum here regarding a similar issue, and it seems the conclusion on there was that precision-recall curves are better-suited for data sets where there is a large negative ...
1
vote
1answer
37 views

Improving F1 scores using models with good precision and recall

I have a highly imbalanced dataset (0.21 percent positives, rest negatives) for which I am trying to build a classifier. I tried to improve the F1 scores using hyperparameter tuning but in all the ...
1
vote
1answer
256 views

Should I balance the classifier train/test set, if metrics is Precision/Recall (F1 score)?

I want to train a classifier on an unbalanced data set. Proportions of classes C0/C1 are 65/35. Importantly, the success metrics is F1_score. In other words, the proper classification of class 1 (...
1
vote
0answers
12 views

How to verify that one ML-classifier is better then the other using the same train and test data set without cross-validation?

I have compared 5 methods (ML classifiers) on the same data set. These methods are 5 different types of neural networks. Each is trained on the training set and evaluated with precision, recall and f1-...
0
votes
1answer
15 views

All of my AU-PRC values are the same, is there something wrong with my code or models?

I have been doing some training of basic models for a certain binary outcome, and most of the training has been on optimizing the AUC. But when I plot the precision recall curves, I get essentially ...
0
votes
1answer
142 views

What is the difference between Top-1 Accuracy and Recall?

I'm doing research on image classification, and I don't understand the difference between Top-1 Accuracy and Recall. Are they ...
1
vote
0answers
32 views

Precision-recall curve for highly imbalanced multi-class classification task

I would like to measure the performance of my network on a multi-class classification problem and wanted to use the precision-recall curve. I have four classes, of which three are extremely ...
0
votes
0answers
18 views

How to calculate IOU for a single image with multiple prediction boxes?

Lets say I have mask-rcnn model that I want to get its performance. I have a image with 1 gt box and for example 3 predictions. One correct and 2 miss matches. How do I calculate IOU for that image ?
0
votes
1answer
27 views

Real vs True Positives

Wikipedia defines TPR (True Positive Rate) as $\frac{\text{TP}}{P}$ where: $\text{TP}$ = # of true positives $\text{P}$ = # of real positives This confuses me. I thought: $\text{TP}$ is supposed ...

1
2 3 4 5
8