Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
1
vote
0answers
6 views

How to find the optimal threshold for the Weighted f1 score in a binary classification problem

I know how to find the optimal threshold for the standard f1 score but do not know how to do so for the weighted f1 score with the sklearn library. Sklearn provides a way to compute the weighted f1-...
0
votes
0answers
11 views

Finding the statistical significance of a Matthews Correlation Coefficient (MCC) in binary classification (what is a good MCC)?

I have a very imbalanced dataset. Thus, I am using MCC to evaluate the performance of various ML algorithms. It appears that literature is entirely lacking in ways to evaluate how good an MCC score is....
0
votes
0answers
7 views

Why there is no tradeoff between precision and recall in this KNN example?

I thought that there would be a trade-off between precision and recall. But that doesn't always seem the case. In the following example, sometimes precision and recall both decrease, and sometime both ...
1
vote
0answers
12 views

Incorporating per-class-accuracy penalties into Deep Learning

Typically when training Neural Networks for image classification, many people use SGD with weight-decay as a penalty term. The loss that is minimized corresponds to the misclassification and state of ...
0
votes
0answers
13 views

Low model performance for an imbalanced data, is there any hope to improve the metrics?

I am working with an imbalanced data: 70k:0 and 1K:1 with 12 features. I would like to perform classification to choose the important features. So far, I have done under-sampling, over-sampling, ...
0
votes
0answers
11 views

Can you do hyperparameter tuning using a PR curve? Moreover, would this still be considered a “PR curve”?

I am creating a graphical representation of PR results across various hyperparameter changes in an imbalanced dataset (model used was an SVM). I'm wondering if one would still consider this a "PR ...
0
votes
1answer
13 views

imbalanced classes: ROC_AUC vs Precision_Recall AUC

I am dealing with a highly imbalanced classes problem. Accuracy is of course not a good performance metric in such cases, So I want to calculate either ROC AUC sore ...
0
votes
0answers
12 views

Is it fair to compare two methods based on the area under the PR curve if the baseline precisions are different?

For two methods with different baseline precisions, is it fair to say one method is better than the other because the area under the PR curve is better? Or should we correct for the baseline ...
0
votes
1answer
25 views

Why macro F1 measure can't be calculated from macro precision and recall?

I'm interested in calculating macro f1-score by macro precision and recall manually. But the results aren't equal. What is the difference in the final formula between f1 and f1_new in code? Would you ...
0
votes
0answers
17 views

For imbalanced datasets, is it necessary to use undersampling or upsampling if one evaluates the performance of ML classifier using PR curves?

Previously, I would have assumed evaluating the classification performance of Decision tree and SVM with a PR curve would obviate the need for under/over-sampling since it doesn't evaluate the true ...
0
votes
0answers
36 views

Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?

On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
0
votes
0answers
11 views

ROC convex hull - realisable classifiers

In the paper Realisable Classifiers Theorem 1 shows that there exists a realisable classifier r_i which lies on a line L_ab ...
0
votes
1answer
46 views

Is it possible to get low AUC score but high Precision and Recall?

I am doing classification on a fairly imbalanced dataset (about 1:2 ratio). I have so far so far tried lasso and logistic regression. I didn't downsample the dataset because the sample size is low (...
0
votes
0answers
6 views

Can I conclude from object detection metrics on temporal behavior of a detection system?

Assuming I know typical object detection metrics like precision, recall etc. for a certain class, is there anyway to conclude on the probability that an object of that class which is present in ...
0
votes
1answer
27 views

Perfect recall but moderate precision due to imbalance?

I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended ...
1
vote
0answers
16 views

How to interpret the Precision and Recall curve in-sample vs out-of-sample

I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best ...
0
votes
2answers
45 views

Is it ok a threshold of 0?

I am dealing with a classification problem with a dataset containing 60k rows: 69k are negative class, and 1k is positive. I trained my models and I obtained the confusion matrices with a threshold of ...
0
votes
1answer
37 views

What value for recall implies a logistic regression model is good?

I'm studying logistic regression using Python and about metrics to have a good model, I know this three: accuracy, precision and recall. In the same way, I was studying using a dataset about ads in ...
0
votes
0answers
46 views

Dummy Classifier - No Skill Line on a Binary Classification Precision Recall Curve Using a Stratified Approach - How Calculated?

I am doing binary classification on a highly imbalanced data set where the positive value is about 1.6% of the dataset. I am training and running models but I want to compare against a dummy ...
0
votes
0answers
55 views

COCO evaluation - Negative values on AP and AR

I am trying to evaluate one of the models using COCO dataset metrics. For some of the precision metrics I am getting results that I can comprehend. For example:Average Precision (AP) @[ IoU=0.50:0.95 ...
0
votes
0answers
13 views

R: cannot find precision, recall and f-measured

I ran a logistic model, and want to know calculate precision, recall and f-measured from confusion matrix. Its accuracy is 0.995. ...
0
votes
1answer
27 views

Precision and Recall With A Binary Classifier

I want to calculate the precision and recall of my results, however, I'm a little confused on how to create the confusion matrix. I have a binary response either yes or no from my participants, ...
0
votes
0answers
6 views

Performance Metrics on Within Subjects Study

I ran a simple yes or no within-subject experiment. i.e there were 4 conditions, and each participant answered a single question for a subset of images per condition. My question is how would I ...
0
votes
1answer
29 views

What is the average precision in the case of no positives for a given category in the context of object detection

In attempting to calculate the average precision of an object detection model, I am wondering about an edge case. Suppose at evaluation time that for a given category, that no detections of that ...
0
votes
0answers
21 views

P-value for precision recall curve significance

I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a ...
0
votes
0answers
35 views

Interpretation of Precision and Recall from Object detection API?

In given picture I have precision and recall value -1.000. What does this signify? Could someone please help me to interpret this results? Furthermore Can I calculate F1 score for this average ...
0
votes
0answers
26 views

Is the PR AUC invariant under label flip?

The ROC-AUC curve is invariant under a flip of the labels. I don't know if it's a famous result, so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
0
votes
0answers
17 views

Does it make sense to mention F1 score on multiclassification problem?

I have two questions that came to my mind while reading some comments on a multiclassification problem: I read that models were evaluated using F1 score (which is a combination of Precision and ...
2
votes
2answers
235 views

Weighting common performance metrics by classification outcomes?

Cost-sensitive classification metrics are somewhat common (whereby correctly predicted items are weighted to 0 and misclassified outcomes are weighted according to their specific cost). Some examples ...
1
vote
0answers
38 views

Statistical test for comparing Precision-Recall curves

When evaluating models' performance using Receiver Operator Characteristic (ROC) curves, DeLong's test can be used to assess whether two curves are significantly different. Is there a statistical ...
0
votes
0answers
83 views

Can we use precision and recall in regression?

Precision is the ratio of true positive to all the positive results whereas recall is the ratio of true positives to all relevant results. As we know that precision and recall are classification ...
0
votes
1answer
15 views

How does recall help precision overcome “length-related problems?”

I'm reading the paper on the Bilingual Evaluation Understudy (BLEU) metric BLEU: A Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002) and had a question regarding a quote ...
0
votes
0answers
9 views

What is the right way to take a PR AUC average of a set of binary classifiers?

I have a multilabel classification problem. I am choosing to treat it as a set of independent binary classifiers. Each label has its corresponding skewness which can range from 1% to 20%. Suppose for ...
0
votes
0answers
18 views

I wonder why the precision of 1 class is much lower on the test set relative to the training set?

I used the LightGBM algorithm to predict the outcome of tennis matches. Next I made two confusion matrices, one for the training set and one for the test set. I calculated stats as precision, recall, ...
0
votes
0answers
21 views

High precision, but low recall on training data. What could be wrong?

I am training RNN for timeseries binary classification. I observed that network has high precision, but low recall on both training and testing data. I tried multiple architectures, but same problem ...
0
votes
0answers
6 views

Is it possible to identify which features contribute to the precision and recall for each label in sklearn's classification report output?

Below is an example of classification report output by sklearn. Is there an easy way to identify which features contribute to the precision and recall improvement for each label? I can start with ...
1
vote
0answers
42 views

Precision-Recall Curve and Area under Precision-Recall Curve (AUC)

I created model (logistic regression) and now trying to create Precision-Recall plot and calculate area under Precision-Recall Plot. I'd like to note that this model is defective: ...
0
votes
0answers
19 views

Appropriate naive benchmark for class recall in binary classification for unbalanced dataset

I have an unbalanced dataset with 3969 rows of customer data. The labels are whether or not they subscribed for a loan (yes or no). There are 3618 no cases (91.2%) and 351 yes cases (8.8%). I am more ...
0
votes
0answers
22 views

Choosing between Average Precision and Precision for Classification Evaluation Metric

I am working on a classification problem in which I will be classifying between fraudulent and non fraudulent transactions. I will be detecting 100 transactions having the highest probability of ...
0
votes
0answers
23 views

Interpreting a precision recall curve

I am working on my ms thesis about predicting skin cancer I have these PR curves 1 - They start from 1.0 but they don't end at 1.0, is this a problem? 2 - SVM falls from 1.0 to 0.0 immediately and ...
3
votes
1answer
80 views

Why we use precision/recall in binary classification but sensitivity(=recall)/specificity in medicine?

Sensitivity=recall is used in both fields, but the second metric is different. Why? Both tasks (classification and medicine) look same - data has two classes and we do some predictions on it and want ...
1
vote
0answers
125 views

Precision-Recall Curve Intuition for Multi-Class Classification

I am running a CNN image multi-class classification model with Keras/Tensorflow and have established about a 90% overall accuracy with my best model trial. I have 10 unique classes I am trying to ...
0
votes
0answers
19 views

When to stop training a binary classifier when precision-recall curve and logits output is of interest

I have a binary classifier (DL model) and a balanced dataset (50-50 for the two classes, in both training and testing data). Let's say I want to optimize for precision. I can adjust my positive label ...
0
votes
1answer
236 views

how to find mean average precision of object detection algorithms

To start with, I would like to mention another question which was asked in a better way. But my problem differs. pseudo code for the algorithms I have four different object detection algorithms which ...
1
vote
0answers
100 views

Can precision and recall of both classes in a binary classification be compared?

While I was going through the question How to interpret classification report of scikit-learn? with the following metrics - I saw following claim (in the accepted answer) - you cannot compare the ...
3
votes
2answers
218 views

Disadvantage of precision at k

Suppose 10 documents were retrieved (rectangle with black color is relevant document). In the following table, Precision @ k is calculated. P@10 or "Precision at 10" corresponds to the ...
0
votes
0answers
29 views

Imbalanced classification what is good F1 score?

For imbalanced classification (say 85:15), what is good value of F1 score? An answer https://stats.stackexchange.com/a/217343/285091 says "Experiments indicate that the sweet spot here is around ...
0
votes
0answers
8 views

Model performance: accuracy equals recall for all experiments

I am working on a multiclass imbalanced dataset set. I would like to evaluate the model performance, but I cannot understand why the model accuracy is equal to the recall for all experiments. Class ...
2
votes
1answer
174 views

Starting point of the PR-curve and the AUCPR value for an ideal classifier

I have two questions about the PR-curve: What is the starting point of the PR-curve ? I mean the point which corresponds to the highest possible threshold (i.e. when all scores are below this ...
0
votes
1answer
32 views

imbalanced learning: precision vs recall trade-off

Working on a multi-class problem (five classes) for which the dataset is highly imbalanced (two classes with less than 2% samples). Which metric between precision ...

1
2 3 4 5
9