Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Why PR score is down when balanced accuracy is good?

I just read this discussion here and here. I have a dataset of 977 records where class proportion is 77:23. My balanced accuracy is 75.5, ...
user avatar
  • 1,758
0 votes
0 answers
10 views

Area Under Precision-Recall and Area Under ROC curve for different amount of observations

I am doing a research and thus comparing some algorithms for binary classification. Worth to mention that, the data set is highly imbalanced i.e., the minority class is only 0.2%. Notation: Area Under ...
user avatar
  • 1
0 votes
0 answers
13 views

Cases where specificity can be very high and precision very low simultaneously?

I was trying to understand the difference between a ROC curve and a PR curve by reading this page: ROC vs precision-and-recall curves. A quote from top voted the answer: Interestingly, by Bayes' ...
user avatar
  • 869
1 vote
1 answer
32 views

Logistic Regression: What is the value for precision when recall (true positive rate) is 0?

A quick overview of definitions before I get into the question: True Positive (TP): An actual positive that the model classified as positive False Positive (FP): An actual negative that the model ...
user avatar
  • 211
0 votes
0 answers
14 views

Difference between balanced_accuracy score and macro_averaged recall

I understand balanced_accuracy_score metrics metric is recommended as against accuracy_score in imbalanced learning. But one thing I find strange is this measure is always equal to the macro-averaged <...
user avatar
  • 231
0 votes
1 answer
31 views

Why True Positives equals to True Negatives while calculating micro F1 or accuracy for multi-class single label classification?

I was searching what is the difference between micro F1 and accuracy since sklearn classification_report shows accuracy in place of micro F1. I found multiple ...
user avatar
  • 121
0 votes
0 answers
15 views

Does a single F-score make sense for multiclass classification?

I'm newish to ML and trying to compute some performance metrics for a K-means classifier. I've constructed the following confusion matrix, and computed the per-class precision, recall, and F-scores (...
user avatar
0 votes
0 answers
5 views

Why does sklearn list weighted precision and micro precision separately if they are the same thing?

This post explains that micro precision is the same as weighted precision. (And the logic applies to recall and f-score as well.) So why does sklearn.metrics list micro and weighted as separate ...
user avatar
  • 55
0 votes
1 answer
60 views

Average of precision and recall

A machine learning model is outputting precision and recall for a two-class classification problem (0 and 1) like this: ...
user avatar
  • 101
0 votes
0 answers
70 views

Precision-Recall curve and AUC in multi-class problems using R

I'm trying to evaluate a RandomForest model for multi-class classification using the Area Under Precision-Recall Curve. I need to plot the PR curve of each class and the micro and macro average and ...
user avatar
  • 113
1 vote
1 answer
22 views

Estimate sample size from a variable population

Context: I want to measure accuracy, precision, and recall for individual raters. Each rater completes a variable amount of labels, for ex. rater A may complete 500 in a given time period while rater ...
user avatar
2 votes
2 answers
378 views

Logistic regression metric

I am interested to understand in which scenarios person should use sensitivity, specificity, and when should person opt for precision recall. On a high level I understand for a balanced data set we ...
user avatar
2 votes
1 answer
34 views

When comparing classifiers on different datasets with different prevalences, is it valid to calculate prevalence-adjusted PPV?

Scenario: comparison of 2 different binary classifiers Both classifiers report sensitivity and specificity and number actually positive (P), but classifier 1 is tested on a dataset with prevalence 20%,...
user avatar
0 votes
1 answer
53 views

Accuracy, Recall (sensitivity), Specificity confusion

I came up with this paper with 765 citations! On page 2, it expresses an equation relating these metrics: Accuracy sensitivity (a.k.a Recall). specificity (a.k.a 1 - type1 error rate) prevalence (...
user avatar
  • 325
1 vote
0 answers
69 views

PRAUC curve Interpretation on imbalanced data

My training data is undersampled to a positive to negative ratio of 5%. I observe a PRAUC of .4 on my training data. When I test the model on real-world data where the positive to negative ratio is .5%...
user avatar
  • 21
1 vote
0 answers
37 views

Calculating overall Precision and Recall, one versus all

I've got several confusion matrices, all of binary classification (negative, positive). I would like to get general scores of all the matrices combine. Problem is, that the data is not balanced at all....
user avatar
  • 11
1 vote
0 answers
23 views

Questions about mAP results from YOLOv2 paper

In the YOLOv2 paper, is the mAP metric displayed in Table 3 calculated in the same way as the metric in the column '0.5' from Table 5?
user avatar
  • 873
2 votes
0 answers
89 views

Why does precision_recall_curve() return similar but not equal values than confusion matrix?

INTRO: I wrote a very simple machine learning project which classifies numbers based on the minst dataset: ...
user avatar
0 votes
1 answer
37 views

How to practically calculate the accuracy of each class in muliclass classification problem?

I have the following confusion matrix: ...
user avatar
2 votes
1 answer
226 views

Relationship between Recall, TPR, FPR and Precision

Can Precision and Recall be used to Generate TPR or FPR? In other words, is there any formula that relates the following Evaluation metrics? True Positive Rate (TPR) with either Precision or Recall (...
user avatar
  • 21
1 vote
0 answers
78 views

Area under the lift curve as a model performance metric?

In the context of machine learning, for (pre-threshold) model performance metrics, I keep AUPRC and AUROC as my go-to, and then I consider things like a lift curve or cumulative gains chart as a ...
user avatar
  • 206
2 votes
2 answers
258 views

How can accuracy be greater than my precision, recall and F-Score metrics?

I have trained two models to detect gestures using ambient light and solar panels. I am now testing the two models in different light scenarios. I have a Convolutional Neural Network model that ...
user avatar
  • 21
1 vote
0 answers
30 views

Is this method based on Wilson score intervals a valid method of calculating the error on the area under a precision-recall curve?

Suppose I have a set of continuous prediction score. I vary my decision threshold to produce a set of precision and recall scores, which I denote as P and ...
user avatar
0 votes
0 answers
10 views

Reporting multiclass macro R/P/F1 when classes are absent from real world data

I have developed a hierarchical classification framework for my current project using a 'local classifier per parent node approach' (as described in https://link.springer.com/article/10.1007/s10618-...
user avatar
1 vote
1 answer
143 views

Interpreting NaN values for precision in Confusion Matrix

Please refer to the confusion matrix here: https://imgur.com/a/Iq1epre Would I get precision values of NaN because of 0/0 in the right most columns? Is that even possible? How should I interpret this? ...
user avatar
  • 47
1 vote
1 answer
71 views

How to obtain conditional use accuracy equality with communities with different real positive rates?

The short version is that I would like to know what the confusion matrices (numbers of true positives, false positives, true negatives, and false negatives) should be to achieve conditional use ...
user avatar
1 vote
1 answer
36 views

How to calculate F1, Precision, and Recall for Multi-Label Multi-Classification

I have a predictive model as follows Sample1 Sample2 Sample3 Sample4 Red Yellow Blue Green White Black Orange 65 21 55 40 0 0 1 0 1 0 0 31 40 44 30 0 0 0 0 0 0 0 33 44 56 66 1 0 0 1 0 0 1 63 77 ...
user avatar
  • 249
0 votes
1 answer
55 views

Can I conclude that the classifier is always good when Precision-Recall Curve above the baseline?

I used logistic regression for highly imbalaned data (1=0.6% , 0=99.4%) Since PR curves are sensitive to imbalance, so i used it, but I don't know how to interpret graph appropriately. This is PR-...
user avatar
1 vote
0 answers
26 views

Why do we have a Neyman-Pearson lemma for type one error but not an analogous one for precision and recall of a classifier?

We fix alpha to .05 in statistical testing. We never fix the precision or recall of a test. There's no decision theory lemma that guarantees we can find a decision region with fixed precision or ...
user avatar
  • 224
0 votes
0 answers
13 views

Testing a churn model trained on fix period data

A Churn model that needs to predict the customers more likely to churn after 9 months (from the executon time),the model was trained on customer's (retail bank) transactions/ loans 3 years data ...
user avatar
0 votes
0 answers
23 views

Custom metric or already exist an metric to this problem? Accuracy by ID

at my company we are working with the "recall" metric in a specific problem, however this metric do not reflect the results that we would like to achieve. Take a look at below table. We ...
user avatar
1 vote
1 answer
55 views

Using individual predictions from binary classification model to predict class proportions in group

Suppose I have a group of users of a paying app and I want to predict each month the users that are not going to renew their subscription. This is called churn rate. To do that I create a binary ...
user avatar
  • 672
0 votes
0 answers
40 views

A question about precision

We have 2 Classification models (Random forest on balanced Data Set), the first one classify a Bank's client as a Churner (closing his account) or an active client, the second one classify a Bank's ...
user avatar
0 votes
0 answers
134 views

Precision-Recall curves with multiclass classifier

I would like to plot the PR curves for a multiclass classifier (e.g. 3 classes). In the documentation it states that multiclass is not supported, and instead a series of one vs all classifiers are ...
user avatar
  • 1
0 votes
1 answer
67 views

High Precision and High Recall issue- Random Forest Classification

I am building a classification model using Random Forest technique using GridSearchCV. The target variable is binary where 1 is 7.5% of total population. I have used several values of GridSearch ...
user avatar
0 votes
0 answers
12 views

How should I calculate recall and precision on unbalanced data?

I have a model that classifies people into three segments (X, Y, Z). The model says that there are 150 of X, 250 of Y, and 5000 of Z. However, I don't know for certain that is the overall prevalence ...
user avatar
  • 156
0 votes
0 answers
83 views

Comparing two datasets with Average Precision/Precision Recall Curve

When comparing performances of classifiers between two different datasets, I use the average precision metric (the datasets are very imbalanced and thus ROC or just Precision unpreferable as was ...
user avatar
12 votes
4 answers
2k views

Why is ROC insensitive to class distributions?

I am confused over why ROC is invariance under class distribution described in the paper An Introduction to ROC analysis. I cannot understand the example on why the proportion of positive to negative ...
user avatar
  • 533
0 votes
1 answer
9 views

Measuring how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as 10%, 25%,etc. What I want to show is how ...
user avatar
1 vote
1 answer
26 views

How to measure how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as $10\%$, $25\%$ ,etc. What I want to show ...
user avatar
1 vote
1 answer
46 views

F1 Score is giving good value in imbalanced dataset

If I have an imbalanced dataset that consists of 90% positive points and 10% negative points. Now I created a "dumb" model which always predicts every point as a positive point. The ...
user avatar
3 votes
1 answer
230 views

Precision and recall estimated with stratified sampling

I have a population including rare events (let's call them "event A") and I want to evaluate the precision and recall of a new algorithm to detect the rare events. In the actual population, ...
user avatar
  • 1,345
0 votes
1 answer
116 views

Why is it called Sensitivity/Recall and Specificity?

Where do the terms: Sensitivity, Recall and Specificity come from historically? I've been looking for an answer for quite some time but to no avail. I understand the formulae and what they mean but I ...
user avatar
  • 139
0 votes
1 answer
44 views

can reversing a very inaccurate binary classifier give more accurate predictions?

I was wondering if somehow I get a model for binary classification with let us say a 0.30 accuracy, does this means that if I reverse the outcome of the model (ie. swapping 0s and 1s in the ...
user avatar
1 vote
1 answer
74 views

Measuring Precision/Recall on a biased sample

I am working with ML models that predict e.g. whether an email violates some corporate policy or not. In this case, the "positives" are emails that violate the policy, and the number of ...
user avatar
  • 1,345
1 vote
0 answers
92 views

How to calculate p-value to compare AUPRC for two models?

I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
user avatar
  • 35
1 vote
0 answers
39 views

Average Precision vs average F1

Average precision computes the area under the recall-precision curve by the trapezoidal rule (or midpoint rule). However, we could also compute the F1 score for every threshold and then take the ...
user avatar
1 vote
0 answers
22 views

Help with understanding metrics for imbalanced classification

I am trying to train a neural network to classify chest X-ray scans as my final MSc project. I have a dataset of 13808 image, 3616 labelled COVID, 10192 labelled normal, so the ratio of COVID to ...
user avatar
0 votes
0 answers
17 views

Why is Recall increasing as number of iterations in CatBoost is decreasing?

I have a highly imbalanced data. I'm using CatBoost to build a predictive model as all the variables are categorical. The default number of iteration is 1000. In order to make it faster I gave 100 as ...
user avatar
  • 1
1 vote
1 answer
354 views

Calculating Area Under The Precision Recall Curve with multiclass

I'm trying to calculate AUPR and when I was doing it on Datasets which were binary in terms of their classes, I used average_precision_score from sklearn and this has approximately solved my problem. ...
user avatar
  • 33

1
2 3 4 5
9