# Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

416 questions
Filter by
Sorted by
Tagged with
7 views

### Comparing two datasets with Average Precision/Precision Recall Curve

When comparing performances of classifiers between two different datasets, I use the average precision metric (the datasets are very imbalanced and thus ROC or just Precision unpreferable as was ...
1k views

### Why is ROC insensitive to class distributions?

I am confused over why ROC is invariance under class distribution described in the paper An Introduction to ROC analysis. I cannot understand the example on why the proportion of positive to negative ...
7 views

### Measuring how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as 10%, 25%,etc. What I want to show is how ...
23 views

### How to measure how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as $10\%$, $25\%$ ,etc. What I want to show ...
21 views

### F1 Score is giving good value in imbalanced dataset

If I have an imbalanced dataset that consists of 90% positive points and 10% negative points. Now I created a "dumb" model which always predicts every point as a positive point. The ...
44 views

### Precision and recall estimated with stratified sampling

I have a population including rare events (let's call them "event A") and I want to evaluate the precision and recall of a new algorithm to detect the rare events. In the actual population, ...
29 views

### Why is it called Sensitivity/Recall and Specificity?

Where do the terms: Sensitivity, Recall and Specificity come from historically? I've been looking for an answer for quite some time but to no avail. I understand the formulae and what they mean but I ...
34 views

### can reversing a very inaccurate binary classifier give more accurate predictions?

I was wondering if somehow I get a model for binary classification with let us say a 0.30 accuracy, does this means that if I reverse the outcome of the model (ie. swapping 0s and 1s in the ...
47 views

### Measuring Precision/Recall on a biased sample

I am working with ML models that predict e.g. whether an email violates some corporate policy or not. In this case, the "positives" are emails that violate the policy, and the number of ...
29 views

### How to calculate p-value to compare AUPRC for two models?

I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
10 views

### Average Precision vs average F1

Average precision computes the area under the recall-precision curve by the trapezoidal rule (or midpoint rule). However, we could also compute the F1 score for every threshold and then take the ...
20 views

### Help with understanding metrics for imbalanced classification

I am trying to train a neural network to classify chest X-ray scans as my final MSc project. I have a dataset of 13808 image, 3616 labelled COVID, 10192 labelled normal, so the ratio of COVID to ...
11 views

### Why is Recall increasing as number of iterations in CatBoost is decreasing?

I have a highly imbalanced data. I'm using CatBoost to build a predictive model as all the variables are categorical. The default number of iteration is 1000. In order to make it faster I gave 100 as ...
21 views

### Calculating Area Under The Precision Recall Curve with multiclass

I'm trying to calculate AUPR and when I was doing it on Datasets which were binary in terms of their classes, I used average_precision_score from sklearn and this has approximately solved my problem. ...
18 views

### Is it acceptable if my model has a very high amount of true negatives and a precision score of 100%?

I currently have a model that's tasked with multilabel classification with an extra "no answer" option for negative predictions. The metrics that I'm using are precision, recall, and the F1 ...
18 views

### How to interpret high and similar ROC-AUC across models with slightly more marked differences in prAUC? (multiclass classification)

I am trying to compare the performance of different models for a multiclass classification task. The dataset itself has about 50 different categories with a lot of imbalance (the cardinalities of the ...
55 views

### Why use harmonic mean for precision and recall (f1 score) instead of just the product of precision and recall?

General question here, I understand the purpose of using the harmonic mean to generate the f1 score for model evaluation. I'm not exactly sure though why we don't just take the product of precision ...
42 views

### Evaluation metric for time-series anomaly detection

My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
123 views

### On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
56 views

### Multiclassification: precision-recall from scratch vs sklearn

I would like to know if there´s any issue behind using sklearn's precision/recall metric functions and coding up from scratch in ...
14 views

### How to plot ROC and Recall-precision curves for this obtained data

Good afternoon , Assume we have this obtained data : ( I had trained a SOM map from scratch then i used a number of epochs, Each epoch consist of a number of iterations ) : ...
38 views

### Mean Average Precision (MAP) object detection metric

I am reading up on object detection metrics from this github repository. I have a slight confusion regarding the precision x recall curve mentioned under the Metrics heading. It says that The ...
8 views

### Compute F1 estimate variance for a given precision and recall

For a binary classifier with precision $P = \mathbb P (Y =1 | \hat Y = 1)$ and recall $R = \mathbb P(\hat Y = 1 | Y = 1)$ (where $\hat Y$ is the predicted class and $Y$ is the true class) how can I ...
80 views

### Accuracy always equal to recall

Fitting 3 different models on a 5-class imbalanced dataset. The results show model accuracy always being equal to the recall. How can this be possible? ...
58 views

### When should one look at sensitivity vs. specificity instead of precision vs. recall?

The precision vs. recall tradeoff is the most common tradeoff evaluated while developing models, but sensitivity vs. specificity addresses a similar issue. When should one of these pairs of metrics be ...
99 views

### Dip in Precision / Recall Curve

I have the following precision / recall curve. I am not sure why there would be a dip and then growth of precision, I would expect it to step down as the classifier was loosened. If anyone could shed ...
337 views

### How to find the optimal threshold for the Weighted f1 score in a binary classification problem

I know how to find the optimal threshold for the standard f1 score but do not know how to do so for the weighted f1 score with the sklearn library. Sklearn provides a way to compute the weighted f1-...
81 views

### Finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
15 views

### Incorporating per-class-accuracy penalties into Deep Learning

Typically when training Neural Networks for image classification, many people use SGD with weight-decay as a penalty term. The loss that is minimized corresponds to the misclassification and state of ...
18 views

### Low model performance for an imbalanced data, is there any hope to improve the metrics?

I am working with an imbalanced data: 70k:0 and 1K:1 with 12 features. I would like to perform classification to choose the important features. So far, I have done under-sampling, over-sampling, ...
12 views

### Can you do hyperparameter tuning using a PR curve? Moreover, would this still be considered a "PR curve"?

I am creating a graphical representation of PR results across various hyperparameter changes in an imbalanced dataset (model used was an SVM). I'm wondering if one would still consider this a "PR ...
20 views

### imbalanced classes: ROC_AUC vs Precision_Recall AUC

I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
22 views

### Is it fair to compare two methods based on the area under the PR curve if the baseline precisions are different?

For two methods with different baseline precisions, is it fair to say one method is better than the other because the area under the PR curve is better? Or should we correct for the baseline ...
34 views

### Why macro F1 measure can't be calculated from macro precision and recall?

I'm interested in calculating macro f1-score by macro precision and recall manually. But the results aren't equal. What is the difference in the final formula between f1 and f1_new in code? Would you ...
25 views

### For imbalanced datasets, is it necessary to use undersampling or upsampling if one evaluates the performance of ML classifier using PR curves?

Previously, I would have assumed evaluating the classification performance of Decision tree and SVM with a PR curve would obviate the need for under/over-sampling since it doesn't evaluate the true ...
275 views

### Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?

On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
18 views

### ROC convex hull - realisable classifiers

In the paper Realisable Classifiers Theorem 1 shows that there exists a realisable classifier r_i which lies on a line L_ab ...
345 views

### Is it possible to get low AUC score but high Precision and Recall?

I am doing classification on a fairly imbalanced dataset (about 1:2 ratio). I have so far so far tried lasso and logistic regression. I didn't downsample the dataset because the sample size is low (...
7 views

### Can I conclude from object detection metrics on temporal behavior of a detection system?

Assuming I know typical object detection metrics like precision, recall etc. for a certain class, is there anyway to conclude on the probability that an object of that class which is present in ...
33 views

### Perfect recall but moderate precision due to imbalance?

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended ...
18 views

### How to interpret the Precision and Recall curve in-sample vs out-of-sample

I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best ...
85 views

### Is it ok a threshold of 0?

I am dealing with a classification problem with a dataset containing 60k rows: 69k are negative class, and 1k is positive. I trained my models and I obtained the confusion matrices with a threshold of ...
52 views

### What value for recall implies a logistic regression model is good?

I'm studying logistic regression using Python and about metrics to have a good model, I know this three: accuracy, precision and recall. In the same way, I was studying using a dataset about ads in ...
180 views

### Dummy Classifier - No Skill Line on a Binary Classification Precision Recall Curve Using a Stratified Approach - How Calculated?

I am doing binary classification on a highly imbalanced data set where the positive value is about 1.6% of the dataset. I am training and running models but I want to compare against a dummy ...
132 views

### COCO evaluation - Negative values on AP and AR

I am trying to evaluate one of the models using COCO dataset metrics. For some of the precision metrics I am getting results that I can comprehend. For example:Average Precision (AP) @[ IoU=0.50:0.95 ...
15 views

### R: cannot find precision, recall and f-measured

I ran a logistic model, and want to know calculate precision, recall and f-measured from confusion matrix. Its accuracy is 0.995. ...
31 views

### Precision and Recall With A Binary Classifier

I want to calculate the precision and recall of my results, however, I'm a little confused on how to create the confusion matrix. I have a binary response either yes or no from my participants, ...
7 views

### Performance Metrics on Within Subjects Study

I ran a simple yes or no within-subject experiment. i.e there were 4 conditions, and each participant answered a single question for a subset of images per condition. My question is how would I ...