Questions tagged [f1]
a popular criterion for evaluating binary decision algorithms and classification models.
74
questions
20
votes
2
answers
2k
views
Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity
Accuracy, as a KPI for assessing binary classification models, has major drawbacks: Why is accuracy not the best measure for assessing classification models?. The exact same issues also plague the F1 ...
8
votes
2
answers
379
views
Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc
Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem.
How can I calculate the ...
7
votes
1
answer
4k
views
F1 score, PR or ROC curve for regression
Due to my background as a pure biologist, I've been struggling with the comment acquired from a reviewer about the accuracy test used in my regression study. While I stick to MSE, MAE and R2 as the ...
5
votes
2
answers
2k
views
Why does the 'weighted' f1-score result in a score not between precision and recall?
On the F1 score sklearn page there's a section that explains each of the options for the average parameter. Under the weighted option, it says: "it can result in an F-score that is not between ...
5
votes
0
answers
1k
views
What is the difference of "normal" F1 and macro average F1 score with binary classification
Please note that I always talk about binary classification here. I do not speak about multi class classification.
In case of unbalanced binary datasets it is a good practice to use F1 score. While ...
4
votes
1
answer
523
views
If the AUC score is 100 percent can F1 value be 99.94 percent?
If the AUC score is 100 percent can the F1 value be 99.94 percent? I would expect 100 percent, too.
4
votes
2
answers
1k
views
Reporting F1 Scores
I have a question with regard to the proper way to report F1 scores. Say I am comparing two algorithms one with F1 score of 0.71 and the other of 0.82.
Is it correct to say:
"Algorithm 1 obtained an ...
4
votes
1
answer
1k
views
Use F1 or maximum F1 for model comparisons?
I am comparing a ML classifier to a bunch of other benchmark F1 classifiers by F1 scores. By AUPRC, my classifier does worse than other benchmark methods. When I compared F1 score, however, I got a ...
4
votes
1
answer
302
views
Why use harmonic mean for precision and recall (f1 score) instead of just the product of precision and recall?
General question here, I understand the purpose of using the harmonic mean to generate the f1 score for model evaluation. I'm not exactly sure though why we don't just take the product of precision ...
3
votes
1
answer
180
views
Singular beta in the F-beta vs. threshold score?
Consider this plot of the $F_\beta$ score for different values of $\beta$. I have a hard time getting an intuition as to why they intersect at a same point. (Cf. this blog post.) In other words, why ...
3
votes
1
answer
660
views
Why is "balanced accuracy" an arithmetic mean instead of harmonic?
In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
3
votes
1
answer
5k
views
On which set (train/val/test) do people calculate F1 score, precision and recall?
This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
2
votes
1
answer
1k
views
Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score
I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
2
votes
1
answer
1k
views
Are Jaccard score and F1 score monotonically related?
I have compared the rankings obtained by comparing 10+ classifiers with this two metrics:
Jaccard score
F1 score
They show a perfect correlation. This results holds on 50+ datasets. When comparing ...
2
votes
2
answers
232
views
Classifier can predict time series 1 day in advance, but not more. Why?
To ask the question more precisely: when doing Time Series classification, I observe the classifier prediction is good if test data directly follows (in chronology) the train data. But when the train ...
2
votes
1
answer
427
views
When to use f1_macro vs f1_weighted? [closed]
My dataset is imbalanced. The labels are binary. I'm training 2 logistic regression models.
For the first one, I didnt balance the model with class_weight and used ...
2
votes
1
answer
590
views
F1 score macro-average
In this question I'll differentiate by using lower-case for class-wise scores, e.g. prec, rec, f1, which would be vectors, and the aggregate macro-average Prec, Rec, F1. My formulae below are written ...
2
votes
1
answer
69
views
Comparing probability threshold graphs for F1 score for different models
Below are two plots, side-by side, for an imbalanced dataset.
We have a very large imbalanced dataset that we are processing/transforming in different manner. After each transformation, we run an ...
2
votes
1
answer
277
views
How to compute F1-score when there are NA in the vectors?
I am using the f1-score (f1 = 2 * (precision * recall) / (precision + recall) to compute the similarity between two vectors (let's call them actual and pred). There are nevertheless some missing ...
2
votes
1
answer
869
views
How to calculate F1, Precision, and Recall for Multi-Label Multi-Classification
I have a predictive model as follows
Sample1
Sample2
Sample3
Sample4
Red
Yellow
Blue
Green
White
Black
Orange
65
21
55
40
0
0
1
0
1
0
0
31
40
44
30
0
0
0
0
0
0
0
33
44
56
66
1
0
0
1
0
0
1
63
77
...
2
votes
1
answer
207
views
Harmonic is used in F1 score because it is a conservative metric: How does it help being conservative?
I was reading Jurafsky 3rd edition, page 12-13 chapter 4
Can you explain why is it good to weigh more the smaller of the two items namely $\frac{1}{Precision}$ or $\frac{1}{Recall}$?
Here is the link ...
2
votes
1
answer
924
views
F1 weighted vs. Log loss in SciKit learn RandomSearchCV
I am sorry to ask another question regarding this topic but I am still puzzled about the following:
When I use 'F1_weighted' as my scoring argument in a ...
2
votes
1
answer
83
views
Binary classification metrics - Combining sensitivity and specificity?
The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN).
For ...
2
votes
0
answers
237
views
Average Precision vs average F1
Average precision computes the area under the recall-precision curve by the trapezoidal rule (or midpoint rule). However, we could also compute the F1 score for every threshold and then take the ...
2
votes
1
answer
2k
views
Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?
On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
2
votes
2
answers
4k
views
F1-Score in a multilabel classification paper: is macro, weighted or micro F1-used?
I read this paper on a multilabel classification task. The authors evaluate their models on F1-Score but the do not mention if this is the macro, micro or weighted F1-Score.
They only mention:
We ...
1
vote
2
answers
392
views
Combine accuracy, precision, and recall
I am working on a classification problem. Several models are produced and all have accuracy, precision and recall metrics on test data. I need to pick the best model among the alternatives.
What I can ...
1
vote
1
answer
108
views
Model accuracy versus F1
When training a model (classifier) in TensorFlow, an accuracy value is returned. What is the interpretation of an accuracy of, say, ...
1
vote
1
answer
131
views
F1 score for change point detection
I'm trying to evaluate my change point detection algorithm based on F1 score[1] defined as follows
Let $\mathcal{X}$ denote the set of change point locations provided by a detection algorithm and let $...
1
vote
1
answer
188
views
Confidence Interval of the Average of a F1 Score Samples
I have a number of individual F1 score samples and right now I am measuring the average F1 score across this group. However, I would also like to present a confidence interval on it. Its a continuous ...
1
vote
1
answer
465
views
F1 Score is giving good value in imbalanced dataset
If I have an imbalanced dataset that consists of 90% positive points and 10% negative points. Now I created a "dumb" model which always predicts every point as a positive point. The ...
1
vote
1
answer
2k
views
Should I balance the classifier train/test set, if metrics is Precision/Recall (F1 score)?
I want to train a classifier on an unbalanced data set. Proportions of classes C0/C1 are 65/35. Importantly, the success metrics is F1_score. In other words, the proper classification of class 1 (...
1
vote
1
answer
38
views
what is the correct interpretation of precision, recall and F1 in R?
Im using R and i had some cases of NAs for F1 when there is NA for precision and 0 for recall and also when both are 0, i also noticed that with both 0 i had f1 as Nan. So im not sure how to interpret ...
1
vote
1
answer
2k
views
Can F1 score be equal to zero?
As it is mentioned in F1 score Wikipedia that 'F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0'.
What is the worst condition that was mentioned?
1
vote
1
answer
15
views
Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?
Given the following table of predictions vs. actual states:
...
1
vote
2
answers
113
views
Why don't we use the harmonic mean of sensitivity and specificity?
There is this question on the F-1 score, asking why we compute the harmonic mean of precision and recall rather than its arithmetic mean. There were good arguments in the answers in favor of the ...
1
vote
0
answers
55
views
Statistical significance of performance difference in classification models
Is it possible to assign a p-value to the mean performance difference in three classification models? The models use the same data, same random seed, and use 10-fold cross validation. Model A has a ...
1
vote
0
answers
42
views
Optimal metric for training with Class-specific masked input features and imbalanced dataset
I have a classification problem of 8-classes, which are extremely imbalanced. The input dataset consists of sequences, each of length n features, where n = 19. For each of the 8 classes, I have a ...
1
vote
0
answers
100
views
Extending F1 score to classification with uncertainty
I have a classifier that can return yes, no's and "maybe"s.
The maybe indicates that we don't have certainty in a prediction because the data point is is not sufficiently close to it's ...
1
vote
0
answers
748
views
How to describe f1 soft loss mathematically
Normal F1-score using binarized prediction can be described like this:
$$F_1 = \frac{2 \cdot TP }{2 \cdot TP + FP + FN}$$
But in a loss function for a Machine Learning model, you will typically need ...
1
vote
2
answers
516
views
What is the F1 Score for my prediction when all values are negative?
I have built a model that gives me classification of some cases
here is a comparison between Actual and Prediction
...
1
vote
0
answers
19
views
How to verify that one ML-classifier is better then the other using the same train and test data set without cross-validation?
I have compared 5 methods (ML classifiers) on the same data set. These methods are 5 different types of neural networks. Each is trained on the training set and evaluated with precision, recall and f1-...
1
vote
0
answers
2k
views
Maximize F1 Score for an imbalanced data and multi-class classification
I'm dealing with an multiclass classification problem. The data is textual and too imbalanced. I see that the models that i'm building using the character level or word level grams are always giving ...
1
vote
0
answers
1k
views
Issue with training on classification metrics other than accuracy (using R and caret)
I have a binary classification problem with two classes 0 and 1. For training an XGBoost classification model, I apply a balanced data set (50% 0's, 50% 1's).
In reality, 1's are much more abundant ...
1
vote
0
answers
298
views
sklearn f1_score=weighted not matching sample_weight specification
I am trying to figure out exactly what this is doing:
sklearn.metrics.f1_score(y_pred, y_test, sample_weight=[...])
Numerically it simply does not seem to be ...
0
votes
1
answer
163
views
PPV vs Sensitivity, they look the same!
I am looking at the equation PPV and Sensitivity
and I got this
PPV = TP / (TF+FN)
and
Sensitivity = TP / (TF+FN)
Which ...
0
votes
1
answer
114
views
What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize F1 score?
What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize the F1 score? I'm using an xgboost model.
Your help is highly appreciated.
0
votes
1
answer
53
views
Is it possible for a binary classifier to have lower accuracy, macrof1 and binaryf1 but higher ROC AUC? [duplicate]
I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow:
Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC ...
0
votes
1
answer
24
views
What is F1 Score for this diagram?
I have this Venn chart that represent a dataset prediction of Identifying if our products are classified as "A41" standard or not
The Blue Circle represents a Machine Learning Model ...
0
votes
1
answer
210
views
Why True Positives equals to True Negatives while calculating micro F1 or accuracy for multi-class single label classification?
I was searching what is the difference between micro F1 and accuracy since sklearn classification_report shows accuracy in place of micro F1.
I found multiple ...