Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Comparing two datasets with Average Precision/Precision Recall Curve

When comparing performances of classifiers between two different datasets, I use the average precision metric (the datasets are very imbalanced and thus ROC or just Precision unpreferable as was ...
11
votes
4answers
1k views

Why is ROC insensitive to class distributions?

I am confused over why ROC is invariance under class distribution described in the paper An Introduction to ROC analysis. I cannot understand the example on why the proportion of positive to negative ...
0
votes
1answer
7 views

Measuring how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as 10%, 25%,etc. What I want to show is how ...
1
vote
1answer
23 views

How to measure how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as $10\%$, $25\%$ ,etc. What I want to show ...
1
vote
1answer
21 views

F1 Score is giving good value in imbalanced dataset

If I have an imbalanced dataset that consists of 90% positive points and 10% negative points. Now I created a "dumb" model which always predicts every point as a positive point. The ...
2
votes
1answer
44 views

Precision and recall estimated with stratified sampling

I have a population including rare events (let's call them "event A") and I want to evaluate the precision and recall of a new algorithm to detect the rare events. In the actual population, ...
0
votes
1answer
29 views

Why is it called Sensitivity/Recall and Specificity?

Where do the terms: Sensitivity, Recall and Specificity come from historically? I've been looking for an answer for quite some time but to no avail. I understand the formulae and what they mean but I ...
0
votes
1answer
34 views

can reversing a very inaccurate binary classifier give more accurate predictions?

I was wondering if somehow I get a model for binary classification with let us say a 0.30 accuracy, does this means that if I reverse the outcome of the model (ie. swapping 0s and 1s in the ...
1
vote
1answer
47 views

Measuring Precision/Recall on a biased sample

I am working with ML models that predict e.g. whether an email violates some corporate policy or not. In this case, the "positives" are emails that violate the policy, and the number of ...
1
vote
0answers
29 views

How to calculate p-value to compare AUPRC for two models?

I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
0
votes
0answers
10 views

Average Precision vs average F1

Average precision computes the area under the recall-precision curve by the trapezoidal rule (or midpoint rule). However, we could also compute the F1 score for every threshold and then take the ...
1
vote
0answers
20 views

Help with understanding metrics for imbalanced classification

I am trying to train a neural network to classify chest X-ray scans as my final MSc project. I have a dataset of 13808 image, 3616 labelled COVID, 10192 labelled normal, so the ratio of COVID to ...
0
votes
0answers
11 views

Why is Recall increasing as number of iterations in CatBoost is decreasing?

I have a highly imbalanced data. I'm using CatBoost to build a predictive model as all the variables are categorical. The default number of iteration is 1000. In order to make it faster I gave 100 as ...
1
vote
1answer
21 views

Calculating Area Under The Precision Recall Curve with multiclass

I'm trying to calculate AUPR and when I was doing it on Datasets which were binary in terms of their classes, I used average_precision_score from sklearn and this has approximately solved my problem. ...
0
votes
1answer
18 views

Is it acceptable if my model has a very high amount of true negatives and a precision score of 100%?

I currently have a model that's tasked with multilabel classification with an extra "no answer" option for negative predictions. The metrics that I'm using are precision, recall, and the F1 ...
0
votes
0answers
18 views

How to interpret high and similar ROC-AUC across models with slightly more marked differences in prAUC? (multiclass classification)

I am trying to compare the performance of different models for a multiclass classification task. The dataset itself has about 50 different categories with a lot of imbalance (the cardinalities of the ...
3
votes
1answer
55 views

Why use harmonic mean for precision and recall (f1 score) instead of just the product of precision and recall?

General question here, I understand the purpose of using the harmonic mean to generate the f1 score for model evaluation. I'm not exactly sure though why we don't just take the product of precision ...
1
vote
1answer
42 views

Evaluation metric for time-series anomaly detection

My dataset is time-series sensor data and anomaly ratio is between 5% and 6% 1. For time-series anomaly detection evaluation, which one is better, precision/recall/F1 or ROC-AUC ? When empirically ...
1
vote
1answer
123 views

On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
0
votes
1answer
56 views

Multiclassification: precision-recall from scratch vs sklearn

I would like to know if there´s any issue behind using sklearn's precision/recall metric functions and coding up from scratch in ...
0
votes
0answers
14 views

How to plot ROC and Recall-precision curves for this obtained data

Good afternoon , Assume we have this obtained data : ( I had trained a SOM map from scratch then i used a number of epochs, Each epoch consist of a number of iterations ) : ...
1
vote
1answer
38 views

Mean Average Precision (MAP) object detection metric

I am reading up on object detection metrics from this github repository. I have a slight confusion regarding the precision x recall curve mentioned under the Metrics heading. It says that The ...
0
votes
1answer
8 views

Compute F1 estimate variance for a given precision and recall

For a binary classifier with precision $P = \mathbb P (Y =1 | \hat Y = 1)$ and recall $R = \mathbb P(\hat Y = 1 | Y = 1)$ (where $\hat Y$ is the predicted class and $Y$ is the true class) how can I ...
3
votes
1answer
80 views

Accuracy always equal to recall

Fitting 3 different models on a 5-class imbalanced dataset. The results show model accuracy always being equal to the recall. How can this be possible? ...
1
vote
0answers
58 views

When should one look at sensitivity vs. specificity instead of precision vs. recall?

The precision vs. recall tradeoff is the most common tradeoff evaluated while developing models, but sensitivity vs. specificity addresses a similar issue. When should one of these pairs of metrics be ...
1
vote
1answer
99 views

Dip in Precision / Recall Curve

I have the following precision / recall curve. I am not sure why there would be a dip and then growth of precision, I would expect it to step down as the classifier was loosened. If anyone could shed ...
2
votes
1answer
337 views

How to find the optimal threshold for the Weighted f1 score in a binary classification problem

I know how to find the optimal threshold for the standard f1 score but do not know how to do so for the weighted f1 score with the sklearn library. Sklearn provides a way to compute the weighted f1-...
0
votes
0answers
81 views

Finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
1
vote
0answers
15 views

Incorporating per-class-accuracy penalties into Deep Learning

Typically when training Neural Networks for image classification, many people use SGD with weight-decay as a penalty term. The loss that is minimized corresponds to the misclassification and state of ...
0
votes
0answers
18 views

Low model performance for an imbalanced data, is there any hope to improve the metrics?

I am working with an imbalanced data: 70k:0 and 1K:1 with 12 features. I would like to perform classification to choose the important features. So far, I have done under-sampling, over-sampling, ...
0
votes
0answers
12 views

Can you do hyperparameter tuning using a PR curve? Moreover, would this still be considered a "PR curve"?

I am creating a graphical representation of PR results across various hyperparameter changes in an imbalanced dataset (model used was an SVM). I'm wondering if one would still consider this a "PR ...
0
votes
1answer
20 views

imbalanced classes: ROC_AUC vs Precision_Recall AUC

I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
0
votes
0answers
22 views

Is it fair to compare two methods based on the area under the PR curve if the baseline precisions are different?

For two methods with different baseline precisions, is it fair to say one method is better than the other because the area under the PR curve is better? Or should we correct for the baseline ...
0
votes
1answer
34 views

Why macro F1 measure can't be calculated from macro precision and recall?

I'm interested in calculating macro f1-score by macro precision and recall manually. But the results aren't equal. What is the difference in the final formula between f1 and f1_new in code? Would you ...
0
votes
0answers
25 views

For imbalanced datasets, is it necessary to use undersampling or upsampling if one evaluates the performance of ML classifier using PR curves?

Previously, I would have assumed evaluating the classification performance of Decision tree and SVM with a PR curve would obviate the need for under/over-sampling since it doesn't evaluate the true ...
0
votes
0answers
275 views

Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?

On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
0
votes
0answers
18 views

ROC convex hull - realisable classifiers

In the paper Realisable Classifiers Theorem 1 shows that there exists a realisable classifier r_i which lies on a line L_ab ...
0
votes
1answer
345 views

Is it possible to get low AUC score but high Precision and Recall?

I am doing classification on a fairly imbalanced dataset (about 1:2 ratio). I have so far so far tried lasso and logistic regression. I didn't downsample the dataset because the sample size is low (...
0
votes
0answers
7 views

Can I conclude from object detection metrics on temporal behavior of a detection system?

Assuming I know typical object detection metrics like precision, recall etc. for a certain class, is there anyway to conclude on the probability that an object of that class which is present in ...
0
votes
1answer
33 views

Perfect recall but moderate precision due to imbalance?

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended ...
1
vote
0answers
18 views

How to interpret the Precision and Recall curve in-sample vs out-of-sample

I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best ...
0
votes
2answers
85 views

Is it ok a threshold of 0?

I am dealing with a classification problem with a dataset containing 60k rows: 69k are negative class, and 1k is positive. I trained my models and I obtained the confusion matrices with a threshold of ...
0
votes
1answer
52 views

What value for recall implies a logistic regression model is good?

I'm studying logistic regression using Python and about metrics to have a good model, I know this three: accuracy, precision and recall. In the same way, I was studying using a dataset about ads in ...
0
votes
0answers
180 views

Dummy Classifier - No Skill Line on a Binary Classification Precision Recall Curve Using a Stratified Approach - How Calculated?

I am doing binary classification on a highly imbalanced data set where the positive value is about 1.6% of the dataset. I am training and running models but I want to compare against a dummy ...
0
votes
0answers
132 views

COCO evaluation - Negative values on AP and AR

I am trying to evaluate one of the models using COCO dataset metrics. For some of the precision metrics I am getting results that I can comprehend. For example:Average Precision (AP) @[ IoU=0.50:0.95 ...
0
votes
0answers
15 views

R: cannot find precision, recall and f-measured

I ran a logistic model, and want to know calculate precision, recall and f-measured from confusion matrix. Its accuracy is 0.995. ...
0
votes
1answer
31 views

Precision and Recall With A Binary Classifier

I want to calculate the precision and recall of my results, however, I'm a little confused on how to create the confusion matrix. I have a binary response either yes or no from my participants, ...
0
votes
0answers
7 views

Performance Metrics on Within Subjects Study

I ran a simple yes or no within-subject experiment. i.e there were 4 conditions, and each participant answered a single question for a subset of images per condition. My question is how would I ...
0
votes
1answer
227 views

What is the average precision in the case of no positives for a given category in the context of object detection

In attempting to calculate the average precision of an object detection model, I am wondering about an edge case. Suppose at evaluation time that for a given category, that no detections of that ...
0
votes
1answer
131 views

P-value for precision recall curve significance

I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a ...

1
2 3 4 5
9