Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

0
votes
0answers
11 views

How much “ground truth” data do I need to evaluate a heuristic classification approach?

I have a heuristic approach to a de-duplication problem: There are about 300 million unknown data points. I attempt to match each data point with one of about 4 million independently known data ...
0
votes
0answers
9 views

How to find AUC for Precision-recall curve using glmnet in R? [closed]

I am running penalized regression in R on an imbalanced dataset using glmnet. Since glmnet masks all other AUC functions in other packages, Can anybody help on how to find Precision-Recall AUC using ...
0
votes
0answers
3 views

What values should the Interpolated 11-point precision curve, measured at k, have when P@k = 0?

I'm evaluating a few retrieval models using the 11-point Interpolated precision x recall curve (here is a description of the protocol I've been using: https://nlp.stanford.edu/IR-book/html/htmledition/...
0
votes
0answers
8 views

Which performance metric to use for stratified data? [duplicate]

I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
0answers
16 views

Is recall a relatively meaningless metric in a balanced dataset?

Just looking for a sanity check here. Leaving aside precision, is talking about the recall of a binary classification algorithm sensible where 50% of the cases presented to it are positive and 50% ...
0
votes
0answers
10 views

When do we use precision/recall over accuracy when evaluating a classifier? [duplicate]

When do we use precision/recall over accuracy when evaluating a classifier? What kind of scenarios would mean precision/recall metrics would be better than accuracy?
0
votes
1answer
57 views

Am I allowed to average the list of precision and recall after k-fold cross validation?

I have created a 5-fold cross validation model and used cross_val_score function to calculate the precision and recall of the cross validated model as follows: ...
2
votes
1answer
24 views

Which metric to use in an ordering problem? auPR / ROC / Lift?

I need to order Users from most likely to perform a binary action X in the next n days, to ...
0
votes
1answer
38 views

ROC and PR curves after over/under sampling in Unbalanced datasets

As I understood till now, ROC curves are not a good presentation of unbalanced datasets and PR curves are preferred because ROC curves are not sensitive to false positives. If we now use resample ...
0
votes
1answer
20 views

High precision or High recall [duplicate]

I have a question about precision and recall. Which classifier is better, one with high precision or high recall for medical purposes like finding patients with allergy? Can we use F-measure for ...
0
votes
0answers
19 views

Evaluating binary classifier model. What can say precision, recall etc.? [duplicate]

i'm trying to understand wether my model has good performance or not. I have binary classifier for summarization sentences: important or not (extractive approach) on specific corpus. Dataset is ...
1
vote
1answer
77 views

calculating overall error in k-fold cross validation

when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however ...
2
votes
2answers
41 views

Does precision or recall have more importance? Are they to be considered equivalent measures of accuracy?

Does precision or recall have more importance? Or are they to be considered equivalent measures of accuracy? They can produce different numbers; they refer to slightly different errors. But is it ...
2
votes
0answers
38 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
0
votes
1answer
37 views

The representation of F1-score on the Precision-Recall Curve

Is there a way to project the F1-score on the precision-recall curve for a such binary classifier? Is there a relationship between the area under the precision-recall curve and F1-score? ...
0
votes
0answers
23 views

Understanding model performance

I built a multiclass (and very imbalanced classes) classifier. When evaluating I found an average F1 of .98 and the classifier seems to be working rather well. However, on evaluating the ROC and ...
0
votes
0answers
25 views

Best performance metric for highly imbalanced dataset f1 score vs kappa vs AUROC

I have highly imbalanced data (like fraud detection). I usually use f1 score to evaluate model performance. But I also saw people recommend AUROC and cohen's kappa. I'm seeking expert opinion on what ...
0
votes
2answers
29 views

Plot ROC or PR curves from either the X,Y coordinates (i.e TPR/TNR; or PPV/TPR) or list of predictions (class probabilities)? [closed]

I have a list of X,Y coordinates for plotting both a ROC curve and a PR curve. I also have the data which was used to calculate those coordinates (i.e. a list of individual predictions with binary ...
0
votes
0answers
21 views

which recall value to plot for same precision in PR curve?

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following: $$[False, True, False, True, True, True, False, False],$$ which leads to the following points ...
1
vote
0answers
29 views

Computing recall on classification task

I have done google searches on computing "recall @ X" several times in the last few months, and every page I read gives me the same answer: $\frac{TP}{TP + FN}$ ... but the story I get from my PI and ...
0
votes
0answers
22 views

What a convex Precision-Recall curve means for training dataset?

Situation I have trained a GBDT model(gradient-boosted decision tree, a tree ensemble model) with a training dataset, and when I calculate PR curve on the same training set, it looks convex: For ...
0
votes
0answers
11 views

what happened if we reverse a fraud classification model?

Let us assume we train a classification model on fraud data sets to detect fraud and no fraud. The classification model threshold has high precision and low recall with ROC AUC=0.9, so we will only ...
0
votes
1answer
18 views

Precision and Recall: Valid Combination?

I'm looking at a precision-recall curve for a binary classification task. My precision-recall curve intersects the y-axis (precision) at 60% and the x-axis at 15%. So I get 15% precision at 100% ...
2
votes
0answers
440 views

Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to ...
0
votes
1answer
441 views

Is there any difference between Sensitivity and Recall?

In most of the places, I have found that sensitivity=recall. In terms of the Confusion Matrix, the formula for both of these is the same: $TP/(TP+FN)$. Is there any difference between these two ...
1
vote
0answers
55 views

Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data

I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing). I am performing classification using ...
0
votes
1answer
50 views

Interpreting precision and recall graphs

i have a CNN for sentiment analysis whose precision and recall for validation data over 10 training epochs is (average:macro): The dataset contains more positive samples than negatives.I have ...
1
vote
1answer
84 views

ROC and PR curves interpretation

Having two plots of ROC and PR curves (by scikit-learn) on one dataset raised me a question. The generated Precision-recall plot shows high precision and high recall, that is, low false positive rate ...
0
votes
0answers
12 views

Average Precision or FBeta & Decision Threshold Tuning for Binary Classifier [duplicate]

I'm working with an imbalanced binary classifier data set (3% positive) in sklearn. The cost of a false negative is extremely high so recall is much more important than precision. To baseline my ...
0
votes
0answers
17 views

Which curve comparison should I use to evaluate the performance of a recommender?

I am building a recommender system on the Last.FM dataset (link here) (1,892 users and 17,632 artists and the number of times a particular artist was listened to by a user). Next, the raw dataset was ...
2
votes
0answers
144 views

What does it mean if the ROC AUC is high and the Average Precision is low?

I have a model that produces a high ROC AUC (0.90), but at the same time a low average precision (0.30). From what I've found, I think it might have to do something with imbalanced data (which the ...
0
votes
1answer
17 views

Is there a 'precision' or 'recall' metric for True Negatives?

Noob question.. Accuracy is defined as percentage of predictions that are correct. Accuracy = (True Positives + True Negatives)/(All classifications) Precision is defined as percentage of ...
0
votes
0answers
28 views

Improving cold-start recommend systems

I am using tensorrec (python framework for recommendation, based on tensorflow) to predict a users choice of content, based on the users meta-data. My current accuracy is at about 2,5% *. Since this ...
0
votes
0answers
20 views

Number of Samples - Speaker Verification Model Evaluation - Precision-Recall-F1

When testing my Speaker Verification model, I am calculating Precision-Recall and F1 measure My test is as follow: This is considered as a binary classification problem. Either sample is from my ...
0
votes
1answer
323 views

MAE and Precision for Collaborative Filtering Recommender Systems

I have got a question concerning Recommender Systems and Evaluation Metrics. I tested a few collaborative filtering recommendation algorithms on dataset containing amazon ratings. Here you can see MAE ...
0
votes
1answer
175 views

Q: Possible to optimize for area under the precision-recall curve in glmnet logistic regression?

tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
1
vote
0answers
144 views

Area Under the Precision Recall curve -similar interpretation to AUROC?

I am trying to interpret the AUCPR. Say I have the following Precision-Recall curve. Firstly: It ends at 0.38 on the y-axis because this particular plot has ...
0
votes
1answer
97 views

How do you calculate precision and recall for multiclass classification with only two classes?

I'm trying to predict the gender of a Twitter account using only the profile information like tweet text, description and used colors. I've trained a SVM classifier and then tested dividing the ...
2
votes
0answers
27 views

Classification models: finding name for specific loss function

Below is the linear classifier analogy, where the two lines are the decision boundaries with different thresholds that gives 0 false positive and 0 false negative respectively. A, B, C are sets of ...
0
votes
0answers
216 views

Recall equals to accuracy but different to precision

I've read this question, and basically I'm having the same issue. I'm dealing with a binary classification problem. I'm calculating the precision, recall and f1 using ...
0
votes
0answers
190 views

Comparing precision-recall curves for training and test data set

Referring to these links for some of my assumptions - https://classeval.wordpress.com/introduction/introduction-to-the-precision-recall-plot Does it make sense to plot train and test results on a ...
0
votes
0answers
114 views

How to work with non square confusion matrix?

Situation I have a classification problem with two discrete classes A and B. Every element without exception belongs either to one or to the other. But my classifier has the option of classifying an ...
0
votes
0answers
29 views

Evaluating external quality of clustering algorithm. Corner case: huge number of clusters

I want to evaluate the external quality of a clustering algorithm. In contrast to ordinary clusterings, I have a rather huge number of clusters, but only a few elements per cluster. Most clusters are ...
0
votes
1answer
62 views

Sampling design to measure the performance of a classifier on large set of new data

I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data. Time after this I received a very large set of the same type of data, albeit ...
1
vote
0answers
329 views

Accuracy and F1 score for binary classification

Consider a binary classification, where the precision is 1 for one class and the recall is 1 for the other class. Thus, all false classifications are some elements of class 1 being detected as class 2....
0
votes
0answers
99 views

Optimize F-Score only for certain classes, disregard other classes

I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text. I use a simple naive Bayes classifier (sklearn) to try to predict ratings given a ...
1
vote
0answers
189 views

Iso-F1 curve for Precision-Recall Curve

I'm reading through Sklearn's tutorial on computing precision/recall! I came across this curve called "Iso-F1" curve they are plotting: link. I tried to read their code for generating it, but I can't ...
0
votes
1answer
365 views

How to correctly read a classification report?

Firstly, is there a difference between model performance and it's accuracy? If yes, what exactly? Secondly, what can I interpret from this classification_report of ...
0
votes
0answers
73 views

ROC curve interpretation [duplicate]

In the context of binary classification how do you interpret ROC curve: more precisely: 1) Why the diagonal stand for a random classifier? [Edit] Let's imagine a random classifier: each time he ...
0
votes
0answers
79 views

Is it possible to get different recall and precision after microaveraging?

I am building a Chunk tagger which predicts chunks in the sentences of my test file. The chunks are given as B-CHUNK tag followed by zero or more I-CHUNK tags and punctuations have O tag. After I ...