Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
2
votes
2answers
88 views

Classifier can predict time series 1 day in advance, but not more. Why?

To ask the question more precisely: when doing Time Series classification, I observe the classifier prediction is good if test data directly follows (in chronology) the train data. But when the train ...
0
votes
0answers
10 views

Possible reasons that validation recall is fluctuating across different epochs but the precision is stable?

I know this is not a coding question, but didn't have any idea where can I ask for help on this. I'm training a deep learning model. After each epoch I measure the performance of the model on ...
1
vote
0answers
14 views

ROC Curve for data sets with large negative bias

For context, I've read this forum here regarding a similar issue, and it seems the conclusion on there was that precision-recall curves are better-suited for data sets where there is a large negative ...
1
vote
1answer
20 views

Improving F1 scores using models with good precision and recall

I have a highly imbalanced dataset (0.21 percent positives, rest negatives) for which I am trying to build a classifier. I tried to improve the F1 scores using hyperparameter tuning but in all the ...
0
votes
1answer
25 views

Should I balance the classifier train/test set, if metrics is Precision/Recall (F1 score)?

I want to train a classifier on an unbalanced data set. Proportions of classes C0/C1 are 65/35. Importantly, the success metrics is F1_score. In other words, the proper classification of class 1 (...
1
vote
0answers
11 views

How to verify that one ML-classifier is better then the other using the same train and test data set without cross-validation?

I have compared 5 methods (ML classifiers) on the same data set. These methods are 5 different types of neural networks. Each is trained on the training set and evaluated with precision, recall and f1-...
0
votes
1answer
11 views

All of my AU-PRC values are the same, is there something wrong with my code or models?

I have been doing some training of basic models for a certain binary outcome, and most of the training has been on optimizing the AUC. But when I plot the precision recall curves, I get essentially ...
0
votes
1answer
41 views

What is the difference between Top-1 Accuracy and Recall?

I'm doing research on image classification, and I don't understand the difference between Top-1 Accuracy and Recall. Are they ...
1
vote
0answers
20 views

Precision-recall curve for highly imbalanced multi-class classification task

I would like to measure the performance of my network on a multi-class classification problem and wanted to use the precision-recall curve. I have four classes, of which three are extremely ...
0
votes
0answers
10 views

How to calculate IOU for a single image with multiple prediction boxes?

Lets say I have mask-rcnn model that I want to get its performance. I have a image with 1 gt box and for example 3 predictions. One correct and 2 miss matches. How do I calculate IOU for that image ?
0
votes
1answer
18 views

Real vs True Positives

Wikipedia defines TPR (True Positive Rate) as $\frac{\text{TP}}{P}$ where: $\text{TP}$ = # of true positives $\text{P}$ = # of real positives This confuses me. I thought: $\text{TP}$ is supposed ...
0
votes
0answers
8 views

using “recall” as a metric when doing hyper parameter tuning in python

I have been using Azure ML studio for my prediction tasks. since recall was a more important metric for my project, I could set that in Azure as the metric which should be optimized while training the ...
0
votes
1answer
33 views

how taking multiple tests increases chance of detection of illness if you have it?

I thought of this as I'm reading about covid tests accuracy, and I am thinking how taking multiple tests influences the chance of correctly detecting illness/no illness. So, if I understand this ...
0
votes
0answers
13 views

Improve Precision for highly imbalance binary classification without compromising much on Recall score

Have a highly imbalanced dataset with Yes ( 1915 ) and No ( 545946).Splitted this into two items namely Train,Test. Applied Overampling using ADASYN as shown below and left the test part as it is ( no ...
0
votes
0answers
6 views

Ranking Evaluation Metric to be used in recommender when only one true positive is available

I have historic data of top k(ranked) items recommended to a user from which a user has bought only one item. What evaluation metric should be used in such cases where there is a single true positive ...
0
votes
0answers
12 views

Query on Precision-Recall curve

I have an imbalanced dataset - Kaggle's Porto Seguro insurance dataset. I have applied Random Forest and XGBoost classifiers on the imbalanced dataset, under-sampled dataset and over-sampled dataset. ...
4
votes
1answer
88 views

Main options on how to deal with imbalanced data

As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets: Option 1: Create k-fold Cross-Validation samples randomly (or even better create k-fold ...
0
votes
0answers
10 views

Pairwise external evaluation of clustering with a contingency table

I want to evaluate the clusters pairwise based on this publication: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html (I have the ground truth of the real labels) So ...
0
votes
0answers
21 views

Low recall when positive is the minority class?

I have 2 versions of the same dataset, one which is fully balanced and one in which the positives:negatives is 1:2. In both cases, when I train my SVM classifier I get low recall and quite high ...
0
votes
0answers
11 views

Using pos_weight to improve recall in a multi-class multi-label problem

I have a multi-label classification problem, and so I’ve been using the Pytorch's BCEWithLogitsLoss. I’d like to optimize my model for a higher F2 score, and so want to bias it to have greater recall (...
0
votes
0answers
13 views

What Metric to use for text summarization evaluation with ROUGE

The following graphic shows a comparison between different extractive text summarizer on the same text with one reference-summary. I want to know which summarizer is the best, meaning which one ...
1
vote
1answer
42 views

When is a dataset “too imbalanced” for AUC ROC and PR is preferred?

I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/...
0
votes
0answers
14 views

What is a good PR-AUC and should I undersample time series for rare event detection? [duplicate]

I have a binary classifier for a highly imbalanced multivariate time series. I use an LSTM Network to predict the next time step and use the prediction error to decide whether a data point is an ...
1
vote
1answer
80 views

F1 score macro-average

In this question I'll differentiate by using lower-case for class-wise scores, e.g. prec, rec, f1, which would be vectors, and the aggregate macro-average Prec, Rec, F1. My formulae below are written ...
0
votes
0answers
20 views

Recall becomes 1 during training

I have a deep-learning network, a binary classification problem and I work with regularization. During training recall becomes 1 (after the 4th or so epoch). Now, I interpret that as follows: My ...
1
vote
0answers
43 views

Issue with training on classification metrics other than accuracy (using R and caret)

I have a binary classification problem with two classes 0 and 1. For training an XGBoost classification model, I apply a balanced data set (50% 0's, 50% 1's). In reality, 1's are much more abundant ...
1
vote
1answer
18 views

what is the correct interpretation of precision, recall and F1 in R?

Im using R and i had some cases of NAs for F1 when there is NA for precision and 0 for recall and also when both are 0, i also noticed that with both 0 i had f1 as Nan. So im not sure how to interpret ...
0
votes
1answer
59 views

k-fold cross validation much better than unseen data

this is my first "real" project and I am not understanding a certain behaviour. My dataset spans from 2017 up to today. What I did is cleaning data, getting rid of missing values etc. There are mixed ...
0
votes
0answers
21 views

PR vs ROC AUC curve

I'm working on a model which decides whether a bank transaction is relevant for an acceptant to review or not. This process is now done by hand when someone applies for a loan. Bank transactions are ...
0
votes
1answer
64 views

Which performance metrics for highly imbalanced multiclass dataset?

I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly ...
0
votes
0answers
11 views

Optimizing Precision, Recall, and F Scores By Combining Search Terms

Sorry if this is an easy question to solve but I'm a bit stuck... I have ~900 search terms in a database which I'm comparing to my "gold standard" dataset for precision, recall, and F score. So I ...
1
vote
0answers
26 views

Relationship between AUC of ROC curve and AUC of PR (precision-recall) curve

I know that both the ROC curve and the PR curve can be used to evaluate the performance of a binary classification prediction model, and PR curve is preferred in the case of imbalanced class ...
0
votes
0answers
32 views

Multi-classification: low precision due to imbalanced classes in test data - what to do?

I built a multi-classification model with 3 result classes (XGBoost using R's caret-package): A, B and C. I undersampled my training data - so every class is equally abundant for training. The ...
4
votes
2answers
101 views

Can we validate accuracy using precision and recall?

My apologies in advance if I am skipping some basics. But I know the formula and understanding of how to calculate accuracy, precision and recall. My question is, given the accuracy, can we validate ...
0
votes
0answers
16 views

Sci-kit Learn Beta Score interpretation: How to use the 'beta' and 'average' param correctly?

I am building a model for a class imbalance problem, which I want to be as recall oriented as possible for the minority class. The model I have built uses class weights to penalize the majority class (...
0
votes
1answer
16 views

How to define precision@k in this KDD paper?

I am following this KDD paper trying to learn new things: http://keg.cs.tsinghua.edu.cn/jietang/publications/KDD12-Tang-et-al-Cross-Domain-Collaboration-Recommendation.pdf In the results section they ...
2
votes
1answer
24 views

Precision-Recall representation - last value of the curve

When reading of precision-recall curves and seeing examples, the last point of the curve is always a given value of precision for a recall of 1. I'm a little confused about this. I have a detector ...
2
votes
1answer
29 views

Seeking Suggestion on Image Dataset for Logistic Regression

The problem is based on binary classification (target and interference). I have an image dataset for which I can not use pixel intensities. I can use pixel coordinates only. Nevertheless, CNN will ...
0
votes
0answers
11 views

What does using the probabilities of positive class mean for area under the curve?

In many of the scikit models, there is a predict_proba method that returns a NxM matrix, ...
0
votes
0answers
13 views

Classifiers evaluation

I'm building a classifier which will detect failures in production of lithium-ion batteries. The classifier has to detect failures with >=90% precision. To do that, I'm building different binary ...
1
vote
1answer
36 views

What are benchmarks for precision when working with unbalanced data?

I have a dataset where the positive class is 1.7%, which equates to about 40k positive cases and a total basis of approx 2.5m. What is a realistic precision to achieve for the most likely to cancel? ...
0
votes
1answer
42 views

precision recall AUC or ROC AUC question

I'm working on a project where the 'in the wild' prevalence is a significant imbalance (e.g. minority 4%). However, I was able to collect data in a balanced manner, i.e. 4,000 samples of minority and ...
3
votes
1answer
56 views

Adjusting classification thresholds based on test set predictions

I have a binary classifier that was trained using k-fold cross validation. I then use the model to get predictions on an unseen (held out) test data set. For my specific application, I would like a ...
1
vote
1answer
102 views

What are best practices for choosing the beta for an F-measure score?

There's some discussion on what F-measure means. I understand that the beta parameter determines the weight of recall in the combined score. In specific one answer states that "for good models using ...
1
vote
0answers
40 views

Precision and recall for SVM from Confusion matrix is different from Precision-Recall graph

Coming from Stackoverflow- So, I am creating a SVM model for a highly imbalanced data set and trying to create to calculate F, Pression and recall from the confusion matrix of the model. Confusion ...
2
votes
1answer
84 views

Base rate of accuracy after resampling for classification problems

If I had an imbalanced dataset with 10% positive instances and 90% negative ones, the base rate for accuracy before resampling is 90%. But what about I resampled the data such that I have an equal ...
1
vote
2answers
226 views

ROC and AUC for imbalanced data? [duplicate]

I've some trouble understanding how to interpret the ROC and it's area under the curve for a classification task. In general, the higher the AUC the better the model can classify true as true and ...
0
votes
0answers
29 views

How to properly report confidence from distance/threshold (face detection)

In the context of face matching I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the distances ...
0
votes
0answers
19 views

Evaluate precision and recall results

The following table shows the precision and recall values I obtained for three object detection models. I evaluate the first two models as the following. The target is to find the best object ...
0
votes
0answers
16 views

Can we calculate mean recall and precision

I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number ...

1
2 3 4 5
8