Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

P&R are a way to measure the relevance of set of retrieved instances. Precision is the % of correct instances out of all instances retrieved. Relevance is the % of true instances retrieved. The harmonic mean of P&R is the F1-score. P&R are used in data mining to evaluate classifiers.

0
votes
2answers
16 views

Plot ROC or PR curves from either the X,Y coordinates (i.e TPR/TNR; or PPV/TPR) or list of predictions (class probabilities)? [on hold]

I have a list of X,Y coordinates for plotting both a ROC curve and a PR curve. I also have the data which was used to calculate those coordinates (i.e. a list of individual predictions with binary ...
0
votes
0answers
10 views

which recall value to plot for same precision in PR curve?

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following: $$[False, True, False, True, True, True, False, False],$$ which leads to the following points ...
1
vote
0answers
22 views

Computing recall on classification task

I have done google searches on computing "recall @ X" several times in the last few months, and every page I read gives me the same answer: $\frac{TP}{TP + FN}$ ... but the story I get from my PI and ...
0
votes
0answers
11 views

Cross-fold validation - f1 scores iterations (sudden drop)

I am performing a cross-fold validation in my model to find the best number of iterations to train the model based on f1 scores. For some fold iterations, I find that there is a sudden drop in the F1 ...
0
votes
0answers
8 views

What a convex Precision-Recall curve means for training dataset?

Situation I have trained a GBDT model(gradient-boosted decision tree, a tree ensemble model) with a training dataset, and when I calculate PR curve on the same training set, it looks convex: For ...
0
votes
0answers
9 views

what happened if we reverse a fraud classification model?

Let us assume we train a classification model on fraud data sets to detect fraud and no fraud. The classification model threshold has high precision and low recall with ROC AUC=0.9, so we will only ...
0
votes
1answer
13 views

Precision and Recall: Valid Combination?

I'm looking at a precision-recall curve for a binary classification task. My precision-recall curve intersects the y-axis (precision) at 60% and the x-axis at 15%. So I get 15% precision at 100% ...
2
votes
0answers
83 views

Confidence interval of precision / recall and F1 score

To summarise the predictive power of a classifier for end users, I'm using some metrics. However, as the users input data themselves, the amount of data and class distribution varies a lot. So to ...
0
votes
1answer
24 views

Is there any difference between Sensitivity and Recall?

In most of the places, I have found that sensitivity=recall. In terms of the Confusion Matrix, the formula for both of these is the same: $TP/(TP+FN)$. Is there any difference between these two ...
1
vote
0answers
20 views

Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data

I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing). I am performing classification using ...
0
votes
1answer
37 views

Interpreting precision and recall graphs

i have a CNN for sentiment analysis whose precision and recall for validation data over 10 training epochs is (average:macro): The dataset contains more positive samples than negatives.I have ...
1
vote
1answer
49 views

ROC and PR curves interpretation

Having two plots of ROC and PR curves (by scikit-learn) on one dataset raised me a question. The generated Precision-recall plot shows high precision and high recall, that is, low false positive rate ...
0
votes
0answers
7 views

Average Precision or FBeta & Decision Threshold Tuning for Binary Classifier [duplicate]

I'm working with an imbalanced binary classifier data set (3% positive) in sklearn. The cost of a false negative is extremely high so recall is much more important than precision. To baseline my ...
0
votes
0answers
11 views

Which curve comparison should I use to evaluate the performance of a recommender?

I am building a recommender system on the Last.FM dataset (link here) (1,892 users and 17,632 artists and the number of times a particular artist was listened to by a user). Next, the raw dataset was ...
2
votes
0answers
41 views

What does it mean if the ROC AUC is high and the Average Precision is low?

I have a model that produces a high ROC AUC (0.90), but at the same time a low average precision (0.30). From what I've found, I think it might have to do something with imbalanced data (which the ...
0
votes
1answer
8 views

Is there a 'precision' or 'recall' metric for True Negatives?

Noob question.. Accuracy is defined as percentage of predictions that are correct. Accuracy = (True Positives + True Negatives)/(All classifications) Precision is defined as percentage of ...
0
votes
0answers
20 views

Improving cold-start recommend systems

I am using tensorrec (python framework for recommendation, based on tensorflow) to predict a users choice of content, based on the users meta-data. My current accuracy is at about 2,5% *. Since this ...
0
votes
0answers
13 views

Number of Samples - Speaker Verification Model Evaluation - Precision-Recall-F1

When testing my Speaker Verification model, I am calculating Precision-Recall and F1 measure My test is as follow: This is considered as a binary classification problem. Either sample is from my ...
0
votes
1answer
78 views

MAE and Precision for Collaborative Filtering Recommender Systems

I have got a question concerning Recommender Systems and Evaluation Metrics. I tested a few collaborative filtering recommendation algorithms on dataset containing amazon ratings. Here you can see MAE ...
0
votes
1answer
74 views

Q: Possible to optimize for area under the precision-recall curve in glmnet logistic regression?

tl;dr with the R glmnet package, is it possible to optimize for the area under the precision-recall curve, rather than the area ...
0
votes
0answers
60 views

Area Under the Precision Recall curve -similar interpretation to AUROC?

I am trying to interpret the AUCPR. Say I have the following Precision-Recall curve. Firstly: It ends at 0.38 on the y-axis because this particular plot has ...
0
votes
1answer
39 views

How do you calculate precision and recall for multiclass classification with only two classes?

I'm trying to predict the gender of a Twitter account using only the profile information like tweet text, description and used colors. I've trained a SVM classifier and then tested dividing the ...
2
votes
0answers
27 views

Classification models: finding name for specific loss function

Below is the linear classifier analogy, where the two lines are the decision boundaries with different thresholds that gives 0 false positive and 0 false negative respectively. A, B, C are sets of ...
0
votes
0answers
91 views

Recall equals to accuracy but different to precision

I've read this question, and basically I'm having the same issue. I'm dealing with a binary classification problem. I'm calculating the precision, recall and f1 using ...
0
votes
0answers
53 views

Comparing precision-recall curves for training and test data set

Referring to these links for some of my assumptions - https://classeval.wordpress.com/introduction/introduction-to-the-precision-recall-plot Does it make sense to plot train and test results on a ...
0
votes
0answers
60 views

How to work with non square confusion matrix?

Situation I have a classification problem with two discrete classes A and B. Every element without exception belongs either to one or to the other. But my classifier has the option of classifying an ...
0
votes
0answers
26 views

Evaluating external quality of clustering algorithm. Corner case: huge number of clusters

I want to evaluate the external quality of a clustering algorithm. In contrast to ordinary clusterings, I have a rather huge number of clusters, but only a few elements per cluster. Most clusters are ...
0
votes
1answer
56 views

Sampling design to measure the performance of a classifier on large set of new data

I have an certain labeled data set suitable for a classification task. A classifier was optimized on this labeled data. Time after this I received a very large set of the same type of data, albeit ...
1
vote
0answers
189 views

Accuracy and F1 score for binary classification

Consider a binary classification, where the precision is 1 for one class and the recall is 1 for the other class. Thus, all false classifications are some elements of class 1 being detected as class 2....
0
votes
0answers
35 views

Optimize F-Score only for certain classes, disregard other classes

I have a labeled dataset of product reviews where the label is a rating between 1 and 5 and the review is just text. I use a simple naive Bayes classifier (sklearn) to try to predict ratings given a ...
1
vote
0answers
75 views

Iso-F1 curve for Precision-Recall Curve

I'm reading through Sklearn's tutorial on computing precision/recall! I came across this curve called "Iso-F1" curve they are plotting: link. I tried to read their code for generating it, but I can't ...
0
votes
1answer
104 views

How to correctly read a classification report?

Firstly, is there a difference between model performance and it's accuracy? If yes, what exactly? Secondly, what can I interpret from this classification_report of ...
0
votes
0answers
42 views

ROC curve interpretation [duplicate]

In the context of binary classification how do you interpret ROC curve: more precisely: 1) Why the diagonal stand for a random classifier? [Edit] Let's imagine a random classifier: each time he ...
0
votes
0answers
48 views

Is it possible to get different recall and precision after microaveraging?

I am building a Chunk tagger which predicts chunks in the sentences of my test file. The chunks are given as B-CHUNK tag followed by zero or more I-CHUNK tags and punctuations have O tag. After I ...
0
votes
0answers
11 views

Would model selection using the Precision metric of the positive class make a useful classifier?

Would model selection using only the Precision (PPV) or Average Precision (AP; aka AUC-PR) metric of the positive class only make a useful classifier? For a classifier, where only correct prediction ...
0
votes
0answers
46 views

Difference between using thresholds to classify and using a custom objective function

Suppose we are faced with a binary classification problem. The standard approach seems to be to fit a probability estimator using a loss function like log loss and then afterwards determine the ...
0
votes
0answers
80 views

Interpretation of F1, Precision, Recall on unbalanced, multi-class classification

I've generated some graphs of my ongoing Grid Search on hyper-parameters of a neural network. For each metric (F1, precision, recall), I've using the 'macro' version of each metric as the 3 classes ...
2
votes
0answers
62 views

What is the advantage of using precision and recall metrics in classification?

I understand in cases of imbalanced classes in a dataset, accuracy itself is not the best metric as it can be misleading. But in cases of balanced classes, why is precision and recall good metrics? Or ...
0
votes
1answer
134 views

How do we generate Precision Recall curves when we have two output neurons, each determining the probability of belonging to the same class?

For classification using neural networks, if we have only one output neuron, we can vary the threshold for class labeling and get a Precision Recall (PR) curve. I am using cross entropy loss, after ...
0
votes
0answers
57 views

Using sampling weights in precision, recall, etc

Suppose I take a very non-uniform sample $(x_1,y_1) ..., (x_n, y_n)$ from some population and derive sampling weights $w_1, ..., w_n$ for each sample. Next, suppose I do a logistic regression to ...
1
vote
1answer
81 views

Choosing estimator based on Precision-Recall Curve

Hello Cross Validated Community, I hope you will help with this one. Below you see my Precision-Recall curves after performing 10-fold Stratified Cross Validation on my dataset. Note that my ...
0
votes
0answers
25 views

How to determine I am over fitting my machine learning algorithm and what are other method to evaluate performance of machine learning algorithm?

I am trying to recognize patterns through deep learning. I have data set of 850 images that I split (600 into train and 250 to validation). After I run machine learning algorithm, I get results shown ...
0
votes
0answers
13 views

metric score when ranking information is only available at test time

I am training a machine learning model where each training sample consists of a set of "competitors" and the aim is to predict the winner (there might be more than one winner. I don't care about ...
2
votes
0answers
167 views

Harmonic mean of precision, recall and specificity

I have a system whose performance was characterized in terms of the amount of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). There is a rate that summarizes ...
0
votes
0answers
36 views

Dataset imbalance problems: what is the best way to report the performance of your classifier? [duplicate]

I have a fairly simple question which is answered in many different ways all of the web, but I am having difficulty getting a single straight answer. My question relates to the best way to report a ...
0
votes
0answers
73 views

Interpreting precision vs recall?

I just trained a dataset with a Naive Bayes algorithm and the performance of the model are 64%, 96%, 66% for accuracy, precision and recall. Is it okay to have low accuracy and recall but a slightly ...
0
votes
0answers
238 views

Accuracy increases on decreasing the percentage of training data with stable precision, recall and F-score

I am currently working on a classification problem using tf-idf and Naive Bayes for two classes A and B. I have randomly shuffle the dataset before implementation, and I was experimenting with the ...
0
votes
0answers
192 views

Having trouble balancing the recall and precision of my XGBoost model

The title essentially says it all. Below are some details regarding my data and model. This is the current class distribution within my training set: ...
1
vote
0answers
92 views

Comparing area under the precision recall curve for models trained with different prevalence of the positive class

I am trying to determine the optimal prevalence of the positive class to use when training my models. I have decided to use area under the precision recall curve (AUPRC) as my metric for determining ...