Questions tagged [precision-recall]

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

Filter by
Sorted by
Tagged with
2
votes
1answer
22 views

Base rate of accuracy after resampling for classification problems

If I had an imbalanced dataset with 10% positive instances and 90% negative ones, the base rate for accuracy before resampling is 90%. But what about I resampled the data such that I have an equal ...
0
votes
1answer
14 views

What are best practices for choosing the beta for an F-measure score?

There's some discussion on what F-measure means. I understand that the beta parameter determines the weight of recall in the combined score. beta < 1 lends more ...
1
vote
1answer
562 views

calculating overall error in k-fold cross validation

when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however ...
0
votes
0answers
11 views

Precision and recall for SVM from Confusion matrix is different from Precision-Recall graph

Coming from Stackoverflow- So, I am creating a SVM model for a highly imbalanced data set and trying to create to calculate F, Pression and recall from the confusion matrix of the model. Confusion ...
4
votes
2answers
10k views

How to improve F1 score with skewed classes?

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1. I've tried several classification algorithms, without too much fine tuning, just to get a ...
10
votes
1answer
3k views

How the probability threshold of a classifier can be adjusted in case of multiple classes? [duplicate]

The above is a very simple example of having a probability classifier output for a binary-class case either 0 or 1 based on some probabilities. In addition it is straightforward how you can change the ...
16
votes
3answers
13k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
45
votes
9answers
114k views

How to interpret F-measure values?

I would like to know how to interpret a difference of f-measure values. I know that f-measure is a balanced mean between precision and recall, but I am asking about the practical meaning of a ...
8
votes
1answer
2k views

Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to ...
0
votes
1answer
129 views

How to determine I am over fitting my machine learning algorithm and what are other method to evaluate performance of machine learning algorithm?

I am trying to recognize patterns through deep learning. I have data set of 850 images that I split (600 into train and 250 to validation). After I run machine learning algorithm, I get results shown ...
1
vote
1answer
38 views

Why does the 'weighted' f1-score result in a score not between precision and recall?

On the F1 score sklearn page there's a section that explains each of the options for the average parameter. Under the weighted option, it says: "it can result in an F-score that is not between ...
1
vote
2answers
51 views

ROC and AUC for imbalanced data? [duplicate]

I've some trouble understanding how to interpret the ROC and it's area under the curve for a classification task. In general, the higher the AUC the better the model can classify true as true and ...
16
votes
5answers
29k views

What impact does increasing the training data have on the overall system accuracy?

Can someone summarize for me with possible examples, at what situations increasing the training data improves the overall system? When do we detect that adding more training data could possibly over-...
0
votes
1answer
641 views

How to interpret ROC with crossing curves?

I have created a ROC curve for two classifiers on a two-class classification task: The green curve is from a nearest neighbours classifiers. The red one from a tree boosting. As we can see both ...
0
votes
0answers
21 views

How to properly report confidence from distance/threshold (face detection)

In the context of face matching I have the following histogram: blue bins count the comparison distances for "self matches" (comparing two images of the same person). Orange bins count the distances ...
0
votes
1answer
194 views

How do you calculate precision and recall for multiclass classification with only two classes?

I'm trying to predict the gender of a Twitter account using only the profile information like tweet text, description and used colors. I've trained a SVM classifier and then tested dividing the ...
0
votes
0answers
17 views

Evaluate precision and recall results

The following table shows the precision and recall values I obtained for three object detection models. I evaluate the first two models as the following. The target is to find the best object ...
0
votes
0answers
13 views

Can we calculate mean recall and precision

I'm evaluating the accuracy in detecting objects for my image data set using three deep learning algorithms. I have selected a sample of 30 images. To measure the accuracy, I manually count the number ...
1
vote
1answer
39 views

Can precision and recall of a DNN trained on human-labeled data be higher than precision and recall of the humans who labeled the data?

I was discussing Deep Learning with an academic statistician, who was criticizing the field as "lacking scientific rigor, overhyped and delivering results which are way worse than claimed". In ...
2
votes
1answer
403 views

How to determine a sample size

Given that I have an algorithm that classifies data points as 'true' or 'false'. and I want to estimate its FPR, FNR. It is not a supervised model where I start with a large training set of labeled ...
2
votes
0answers
41 views

How to choose operation point from precision recall curves for multi-label classification

Is there a commonly accepted method for selecting an operating point for a multilabel classifier to optimize for each of these aggregate metrics: micro averaged recall at some minimal acceptable ...
1
vote
0answers
22 views

When is accuracy a good metric (as opposed to precision, recall, F1)? [duplicate]

Suppose you have a perfectly balanced data-set. In which applications is accuracy a good metric? Are there applications where it's preferable to precision, recall, and F1 (all at the same time)?
1
vote
1answer
1k views

Understanding Precision and Recall Results on a Binary Classifier

I know the difference between Precision and Recall metrics in Machine Learning. One optimizes on False Positives and other on False Negative. In Statistics it is called as optimizing on Type I or Type ...
0
votes
0answers
22 views

Accuracy scores in a Deep Learning project

I'm using three pre-trained deep learning models to detect vehicles and count from an image data set. The vehicles belong to one of these classes ['car', 'truck', 'motorcycle', 'bus']. So, for a ...
16
votes
3answers
6k views

ROC vs Precision-recall curves on imbalanced dataset

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset. For example, we have 10 samples in test dataset. 9 samples are positive and 1 is ...
1
vote
0answers
38 views

PR AUC vs ROC AUC for imbalanced data [duplicate]

I am struggling with choosing metric that I will use to compare models performance and hiperparam search. My task is similar to fraud detection. I have found out that many people states PR is better ...
1
vote
1answer
2k views

how to calculate precision, recall, and accuracy, if the label is not binary?

I know how to evaluate the prediction performance if the label is binary, such as the classification label is sick(yes/no). The situation is the label is ...
2
votes
1answer
191 views

Combining Classifiers with different Precision and Recall values

Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is ...
8
votes
4answers
19k views

Average Precision in Object Detection

I'm quite confused as to how I can calculate the AP or mAP values as there seem to be quite a few different methods. I specifically want to get the AP/mAP values for object detection. All I know for ...
2
votes
1answer
37 views

Why does a class weight fraction improve precision compared to undersampling approach where precision drops?

I have an imbalanced data where the ratio between positive to negative samples is 1:3 (positive samples are 3 times higher than negative). For my case it is is important to have a higher precision (...
12
votes
5answers
2k views

Why isn't the sum of Precision and Recall a worthy measure?

What is the best way to explain why $\text{Precision} + \text{Recall}$ is not a good measure, say, compared to F1?
0
votes
1answer
29 views

The precision recall AUCs calculated by two different packages are different?

I used the dataset cars as an example ...
0
votes
0answers
13 views

How come I get 0 recall?

I am doing multinomial naive bayes for the first time; code: ...
0
votes
0answers
39 views

is it possible to form precision-recall curve using one test dataset for my algorithm? [duplicate]

I'm working on knowledge graph, more precisely in natural language processing field. To evaluate the components of my algorithm, it is necessary to be able to classify the good and the poor ...
0
votes
1answer
246 views

How to measure classifier performance on small and skewed sample dataset?

I have a small sample dataset (n=25) that represents the ground truth for a larger set (n=10k). I am doing a classification task and obtain, say, 3 true positives, 20 true negatives, 1 false positive, ...
0
votes
0answers
19 views

What is the standard metric used in recommendation systems to evaluate the rankings?

I was searching for a metric to do this for a while and still could not find. More specifically, my problem is as follows. I have a ranked golden corpus. For example, consider that it looks as ...
0
votes
1answer
198 views

Signs of Overfitting in Precision/Recall Curve

plz look at the following figures. As you cann see the precision is always 100% no matter which threshold (x-axis in logarithmic scale) you set! Also the second figure shows that we have a perfect ...
0
votes
0answers
15 views

Precision & Recall Intepretation

I ran a binary logistical regression: 5 class variables, 1 target Yes/No. The set is imbalanced 77:23 No:Yes. I set model to preditct "Yes". AUC is 0.64 - so not a great predictor. Precision and ...
0
votes
1answer
77 views

How to calculate precision and recall with only one object class?

I have an object detection problem with only one object class. I want to compare the results and thought about using precision and recall. They are defined as follows: $$precision = \frac{TP}{TP + FP} ...
0
votes
0answers
25 views

Precision-Recall Curve for imbalanced dataset and effect of swapping positives and negatives

We are currently trying to evaluate the performance of a binary prediction model, on a dataset which has a majority of positive samples. Having done some research, we read from this paper that pr-auc ...
1
vote
2answers
451 views

Precision and recall in content-based recommender

I have some trouble understanding the concept of using precision and recall to evaluate a content-based recommender. Suppose I want to recommend articles to users. A content-based recommender will ...
11
votes
3answers
4k views

How to choose a good operation point from precision recall curves?

Is there any standard method to determine an "optimal" operation point on a precision recall curve? (i.e., determining the point on the curve that offers a good trade-off between precision and recall) ...
0
votes
0answers
12 views

interpretation of precision and recall when oversampling or undersampling in mlr

I balance my dataset with e.g. cpoUndersample() from mlrCPO Does this balance my test-set as well? This is important because ...
0
votes
0answers
74 views

In XGBoost with a f1_score, is the iteration with a lower or higher score the better iteration?

In the following XGBoost script the output states iteration 0 with score 0.0047 is the best score. I would expect iteration 10 with score 0.01335 to be the better score? Output ...
2
votes
1answer
533 views

Precision, Recall and area under ROC curve as sample size increases

The following is a question from an exam paper on evaluating the performance of search engines. To this day I looked in my text book and literally close to 50 web pages and I can't find one convincing ...
0
votes
0answers
61 views

Why Log loss, AUC and precision & recall change differently when class imbalance problem is tackled?

I have a dataset and I'm working on a binary classification task with it. It has a class imbalance problem where False class versus ...
0
votes
0answers
7 views

How to quantify performance of subclasses?

I have a dataset of $N$ points. Each point $p_n$ has an associated label $l_n$ which is either $0$ or $1$, $n=1$ to $N$. Let $\overline{l}$ be the vector of all $l_n$ stacked together. Say every point ...
2
votes
2answers
261 views

What is the advantage of using precision and recall metrics in classification?

I understand in cases of imbalanced classes in a dataset, accuracy itself is not the best metric as it can be misleading. But in cases of balanced classes, why is precision and recall good metrics? Or ...

1 2 3 4 5 7