Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
-1
votes
0answers
12 views

How to compute 2x2 confusion matrix from large confusion matrix? [closed]

I have 80x80 confusion matrix, I want to compute 2x2 matrix for each of the class. So I can compute precision and recall for each class.
0
votes
0answers
17 views

How to properly report confidence from distance/threshold (face detection)

In the context of face matching I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the distances ...
0
votes
0answers
15 views

Evaluate precision and recall results

The following table shows the precision and recall values I obtained for three object detection models. I evaluate the first two models as the following. The target is to find the best object ...
0
votes
0answers
12 views

Can we calculate mean recall and precision

I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number ...
1
vote
1answer
34 views

Can precision and recall of a DNN trained on human-labeled data be higher than precision and recall of the humans who labeled the data?

I was discussing Deep Learning with an academic statistician, who was criticizing the field as "lacking scientific rigor, overhyped and delivering results which are way worse than claimed". In ...
2
votes
0answers
29 views

How to choose operation point from precision recall curves for multi-label classification

Is there a commonly accepted method for selecting an operating point for a multilabel classifier to optimize for each of these aggregate metrics: micro averaged recall at some minimal acceptable ...
1
vote
0answers
22 views

When is accuracy a good metric (as opposed to precision, recall, F1)? [duplicate]

Suppose you have a perfectly balanced data-set. In which applications is accuracy a good metric? Are there applications where it's preferable to precision, recall, and F1 (all at the same time)?
0
votes
0answers
22 views

Accuracy scores in a Deep Learning project

I'm using three pre-trained deep learning models to detect vehicles and count from an image data set. The vehicles belong to one of these classes ['car', 'truck', 'motorcycle', 'bus']. So, for a ...
1
vote
0answers
33 views

PR AUC vs ROC AUC for imbalanced data [duplicate]

I am struggling with choosing metric that I will use to compare models performance and hiperparam search. My task is similar to fraud detection. I have found out that many people states PR is better ...
2
votes
1answer
28 views

Why does a class weight fraction improve precision compared to undersampling approach where precision drops?

I have an imbalanced data where the ratio between positive to negative samples is 1:3 (positive samples are 3 times higher than negative). For my case it is is important to have a higher precision (...
0
votes
1answer
28 views

The precision recall AUCs calculated by two different packages are different?

I used the dataset cars as an example ...
0
votes
0answers
13 views

How come I get 0 recall?

I am doing multinomial naive bayes for the first time; code: ...
0
votes
0answers
39 views

is it possible to form precision-recall curve using one test dataset for my algorithm? [duplicate]

I'm working on knowledge graph, more precisely in natural language processing field. To evaluate the components of my algorithm, it is necessary to be able to classify the good and the poor ...
0
votes
0answers
16 views

What is the standard metric used in recommendation systems to evaluate the rankings?

I was searching for a metric to do this for a while and still could not find. More specifically, my problem is as follows. I have a ranked golden corpus. For example, consider that it looks as ...
0
votes
0answers
15 views

Precision & Recall Intepretation

I ran a binary logistical regression: 5 class variables, 1 target Yes/No. The set is imbalanced 77:23 No:Yes. I set model to preditct "Yes". AUC is 0.64 - so not a great predictor. Precision and ...
0
votes
1answer
36 views

How to calculate precision and recall with only one object class?

I have an object detection problem with only one object class. I want to compare the results and thought about using precision and recall. They are defined as follows: $$precision = \frac{TP}{TP + FP} ...
0
votes
0answers
24 views

Precision-Recall Curve for imbalanced dataset and effect of swapping positives and negatives

We are currently trying to evaluate the performance of a binary prediction model, on a dataset which has a majority of positive samples. Having done some research, we read from this paper that pr-auc ...
0
votes
0answers
5 views

interpretation of precision and recall when oversampling or undersampling in mlr

I balance my dataset with e.g. cpoUndersample() from mlrCPO Does this balance my test-set as well? This is important because ...
0
votes
0answers
35 views

In XGBoost with a f1_score, is the iteration with a lower or higher score the better iteration?

In the following XGBoost script the output states iteration 0 with score 0.0047 is the best score. I would expect iteration 10 with score 0.01335 to be the better score? Output ...
0
votes
0answers
20 views

Why does the 'weighted' f1-score result in a score not between precision and recall?

On the F1 score sklearn page there's a section that explains each of the options for the average parameter. Under the weighted option, it says: "it can result in an F-score that is not between ...
0
votes
0answers
7 views

How to quantify performance of subclasses?

I have a dataset of $N$ points. Each point $p_n$ has an associated label $l_n$ which is either $0$ or $1$, $n=1$ to $N$. Let $\overline{l}$ be the vector of all $l_n$ stacked together. Say every point ...
0
votes
0answers
43 views

Why Log loss, AUC and precision & recall change differently when class imbalance problem is tackled?

I have a dataset and I'm working on a binary classification task with it. It has a class imbalance problem where False class versus ...
0
votes
1answer
33 views

precision and recall for biased data

here is my assumption input data's labels are 99% true and there are only 1% false(trying to say most of the data are true). My classifier's probability is 0.5. just like unbiased coin flip. in this ...
0
votes
0answers
87 views

Why using mAP (mean Average Precision) instead directly computing AP (Average Precision)

Problem mAP is a metric frequently used in tasks involving ranking. One application of mAP is in object detection, where for each category The predicted bounding box is organized in decreasing order ...
0
votes
0answers
11 views

Compare two precision and recall of two diagnostic tests (with confidence intervals)

I've been asked to compare two diagnostic tests, using precision and recall, and calculate the confidence intervals of the difference between them (or a test of statistical significance would be OK). ...
1
vote
1answer
46 views

Average precision/recall of multiple classifiers on the same dataset?

If I have some N human classifiers that can only predict in terms of 0 or 1 (not probabilistic, and also disregarding their own uncertainty. They either know or they don't), and each yield different ...
0
votes
2answers
49 views

How to train and assess prediction models when data are imperfect (miss some true cases)?

Consider the following example. Suppose your labeled dataset includes images of dogs and you want a computer program for recognizing dogs in images. The problem is that your data are imperfect because ...
0
votes
0answers
172 views

Text classification - High Accuracy, low recall and low precision

I am using fastai to create a text classifier that labels texts as either 0 or 1. My data (number of 1's and 0's) for training is balanced, and I got an accuracy of 85%. To test, I used a new ...
1
vote
2answers
61 views

Precision Vs Recall Curve analysis

I have the following averaged $precision-recall$ curves with $4$ models. Which one is the best?
2
votes
1answer
424 views

Is there a name for this metric: TN / (TN + FN)?

Given a confusion matrix, there's all kind of metrics: Accuracy, Precision, Recall/Sensitivity, Specificity. But I haven't seen any name for the ratio between the TN (True Negative) and the sum of ...
0
votes
0answers
39 views

Understading Overfitting from Precision and Recall scores

Can I understand if my model is overfitting or underfitting from its precision and recall scores that it has on training and test datasets?
3
votes
2answers
167 views

PR AUC < 50% with ROC AUC > 90% - model good or bad?

I understand for highly imbalanced dataset - we need to look for precision-recall vs ROC AUC to better judge the model. My question is what is the range for PR AUC below which the model is bad? My ...
1
vote
0answers
89 views

Understanding Accuracy, Recall and IoU

Working on an image segmenetation problem, I've tackled the following scenario repeated on different images: High Recall and Accuracy (around 99%) Low IoU (around 60%) How is that possible? Recall ...
1
vote
0answers
81 views

Precision, Recall, F1 Score and multiclass confusion matrix

My goal is to evaluate some clustering algorithms. Lets say that I’ve got one set o elements: { 1 ; 2; 3; 4; 5; 6; 7; 8 }. This set was grouped by human into two groups: (1 ; 2; 3; 4; 5; 6; 8) and (7)....
1
vote
1answer
86 views

Can a precision-recall graph have an error bar?

I have only ever seen a precision-recall graph with lines. Could a precision-recall curve have error bars? How would the error bar be computed?
1
vote
0answers
23 views

Is there a simple formula to calculate sample size needed for precision and recall

I have trained a model on 30 towns and this produces good results. I want to apply the model to hundreds of new towns. These are likely similar but I want to measure results and identify any that do ...
0
votes
0answers
25 views

Weighting for precision and recall

I want to integrate the notion of weighting into an evaluation. I am wondering if it is appropriate/correct to calculate precision and recall scores by adding a weighting on true positives, false ...
0
votes
0answers
28 views

Understanding ratio between precision and recall

I have a naive question regarding the ratio of precision and recall. When I build the model, I am able to get precision and recall. Later, I could use this model to make perditions, which is ...
0
votes
0answers
230 views

Precision/Recall against threshold curve not useful in improving model performance

I am doing fraud detection (binary classification) on an unbalanced credit dataset, using SVM. My current precision is 5% while recall is 90%. (I am using undersampling to train and test a model from ...
2
votes
1answer
61 views

Precision-Recall curve interpretation

When given an example confusion matrix: TP = 5000 FP = 1000 FN = 0 ...
2
votes
1answer
44 views

Balanced datasets are almost all predicted negative

Problem I am trying to do sentiment analysis using pretrained word vectors GloVe, which is essentially a look-up table that maps word to a fix-dimension vector. Since GloVe is initially designed to ...
0
votes
1answer
33 views

Does the threshold value of a logistic regression hypothesis has an effect on the accuracy?

It is true that the threshold value of a logistic regression hypothesis has an effect on the Precision/Recall metrics. Suppose you have trained a logistic regression classifier which is outputting $...
0
votes
1answer
59 views

How to interpret precision and recall for multiclass prediction?

I have a few models doing prediction with 4 classes, with the output precision and recall varying with different labels. For example I have (with the class labels being 0, 1, 2, 3 on the x axis): I ...
0
votes
1answer
47 views

Practical interpretation of Precision-Recall AUC

I have a classifier with an AUC (PR) of 0.06 which I will use for a practical interpretation. My test set consists of three months of data with a total of 2,200,000 observations of which 0.03 are ...
0
votes
0answers
47 views

Improveness given a certain AUPRC

I am training a machine learning model (Random Forest) for a multiclass problem (64 classes) in which most of them are highly imbalanced. That's why I am using mainly F1 score for checking the model's ...
0
votes
0answers
78 views

Precision and recall for movielens top n recommendation?

I know the definition. But because the real observed data is very sparse. I often come across the situation where recommended items are disjoint with the test data therefore I have 0 precision and ...
1
vote
1answer
240 views

What is the proper unit for F1? Is it a percentage?

F1-score is defined as $$ F_1 = \frac{2PR}{P+R}, $$ where P is precision [0..1] and R is recall [0..1]. My question is simply, is it right to describe F1 as a percentage? As in "our final F1-score ...
1
vote
1answer
151 views

Is it possible to have recall and precision of 0 while having an area under PR ~0.5?

As the title suggests, I am running a Random Forest classifier using Scala. To evaluate this classifier (and since I am handling highly imbalanced classes), I used the ...
1
vote
1answer
103 views

Optimising recall for multi-label classification?

I'm working on a multi-class multi-label classification problem where text (let's say comments on a website) should be assigned (possibly multiple) labels. There is a neutral (negative) class and ...
0
votes
1answer
169 views

Signs of Overfitting in Precision/Recall Curve

plz look at the following figures. As you cann see the precision is always 100% no matter which threshold (x-axis in logarithmic scale) you set! Also the second figure shows that we have a perfect ...