Questions tagged [f1]

a popular criterion for evaluating binary decision algorithms and classification models.

Filter by
Sorted by
Tagged with
1 vote
1 answer
14 views

Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?

Given the following table of predictions vs. actual states: ...
Bryan's user avatar
  • 1,147
0 votes
0 answers
11 views

Proving stability of F1 metric for a given sample size

Okay so say you have sequence classification problem: extracting entities from conversations. Say one of the labels is CITIES. Say you have calculated P/R/f1/support for CITIES and it looks like this: ...
Winslow Lang's user avatar
2 votes
1 answer
27 views

Binary classification metrics - Combining sensitivity and specificity?

The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN). For ...
usual me's user avatar
  • 1,157
0 votes
0 answers
29 views

Correctness of derivation for binary F1 variance for F1 confidence intervals

I'm developing a python library for confidence intervals for common accuracy metrics, with both analytic and bootstrap computations. Following this paper, I implemented the Macro and Micro F1 scores ...
Jacob G's user avatar
  • 103
0 votes
0 answers
49 views

Inverse Weighted Average F1-Score

I am dealing with a binary classification problem (class 0/1) with class imbalance. Given the vector of predictions, I would like to compute: F1-Score for class 0 F1-Score for class 1 Weighted ...
AngelMarcos's user avatar
1 vote
2 answers
68 views

Why don't we use the harmonic mean of sensitivity and specificity?

There is this question on the F-1 score, asking why we compute the harmonic mean of precision and recall rather than its arithmetic mean. There were good arguments in the answers in favor of the ...
user209974's user avatar
0 votes
0 answers
286 views

F1 score for validation and testing datasets is different

I have the following F1 score function that I use for the model when I train it as part of metrics and as well during prediction: ...
Avv's user avatar
  • 249
5 votes
2 answers
274 views

Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Suppose I have a confusion matrix, or alternatively any one or more of accuracy, sensitivity, specificity, recall, F1 score or friends for a binary classification problem. How can I calculate the ...
Stephan Kolassa's user avatar
16 votes
2 answers
1k views

Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Accuracy, as a KPI for assessing binary classification models, has major drawbacks: Why is accuracy not the best measure for assessing classification models?. The exact same issues also plague the F1 ...
Stephan Kolassa's user avatar
1 vote
0 answers
41 views

Statistical significance of performance difference in classification models

Is it possible to assign a p-value to the mean performance difference in three classification models? The models use the same data, same random seed, and use 10-fold cross validation. Model A has a ...
Adam_G's user avatar
  • 351
0 votes
0 answers
106 views

Relationship between precision-recall curve and Fmax

These two metrics are both usually appropriate for imbalanced classification. Since $$F_{max} = \max{\frac{2\cdot precision\cdot recall}{precision+recall}},$$ I'm guessing Fmax might be somewhat "...
jasperhyp's user avatar
1 vote
1 answer
137 views

Confidence Interval of the Average of a F1 Score Samples

I have a number of individual F1 score samples and right now I am measuring the average F1 score across this group. However, I would also like to present a confidence interval on it. Its a continuous ...
SriK's user avatar
  • 239
3 votes
1 answer
126 views

Singular beta in the F-beta vs. threshold score?

Consider this plot of the $F_\beta$ score for different values of $\beta$. I have a hard time getting an intuition as to why they intersect at a same point. (Cf. this blog post.) In other words, why ...
Tfovid's user avatar
  • 755
1 vote
0 answers
39 views

Optimal metric for training with Class-specific masked input features and imbalanced dataset

I have a classification problem of 8-classes, which are extremely imbalanced. The input dataset consists of sequences, each of length n features, where n = 19. For each of the 8 classes, I have a ...
HATEM EL-AZAB's user avatar
0 votes
0 answers
43 views

Cross-validation and F1 metric

Cross-validation with metrics such as F1 can be implemented in two ways: For each cross-validation split, calculate F1_split on the validation dataset. F1_result = average_by_splits(F1_split) For ...
Arseniy Maryin's user avatar
0 votes
0 answers
46 views

Harmonic mean of false positive and false discovery rates (analogous to F1)

F1 is the harmonic mean of recall (aka sensitivity, or true positive rate, TPR) and precision (aka positive predictive value, PPV). $\text{TPR} = \text{Pr(predicted:Pos | Pos)} =$ TP/P (wikipedia ...
Tomas's user avatar
  • 101
2 votes
1 answer
753 views

Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score

I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like ...
Fredrik's user avatar
  • 725
1 vote
1 answer
97 views

F1 score for change point detection

I'm trying to evaluate my change point detection algorithm based on F1 score[1] defined as follows Let $\mathcal{X}$ denote the set of change point locations provided by a detection algorithm and let $...
thesecond's user avatar
  • 298
2 votes
1 answer
186 views

How to compute F1-score when there are NA in the vectors?

I am using the f1-score (f1 = 2 * (precision * recall) / (precision + recall) to compute the similarity between two vectors (let's call them actual and pred). There are nevertheless some missing ...
Guillon's user avatar
  • 23
1 vote
2 answers
285 views

Combine accuracy, precision, and recall

I am working on a classification problem. Several models are produced and all have accuracy, precision and recall metrics on test data. I need to pick the best model among the alternatives. What I can ...
Emin Ozkan's user avatar
0 votes
0 answers
21 views

How good do accuracy, precision, recall, and F1 have to be to use for flight critical environments?

How good do accuracy, precision, recall, and F1 have to be to use for flight critical environments? Maybe there are rules of thumb, general practices, best practices, lessons learned, etc. Flight ...
JustADude's user avatar
  • 101
0 votes
1 answer
79 views

What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize F1 score?

What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize the F1 score? I'm using an xgboost model. Your help is highly appreciated.
NAS_2339's user avatar
  • 223
3 votes
1 answer
396 views

Why is "balanced accuracy" an arithmetic mean instead of harmonic?

In the F1 score, the harmonic mean of precision (Positive Predictive Value) and sensitivity/recall (True Positive Rate), I understand that we use the harmonic mean in order to penalize extreme values ...
shadowtalker's user avatar
  • 11.8k
2 votes
1 answer
313 views

When to use f1_macro vs f1_weighted? [closed]

My dataset is imbalanced. The labels are binary. I'm training 2 logistic regression models. For the first one, I didnt balance the model with class_weight and used ...
Suvrodip Mukhopadhyay's user avatar
4 votes
1 answer
418 views

If the AUC score is 100 percent can F1 value be 99.94 percent?

If the AUC score is 100 percent can the F1 value be 99.94 percent? I would expect 100 percent, too.
Peter's user avatar
  • 221
0 votes
0 answers
280 views

What to do with 99% F1 score in binary classification?

I've been handed a binary classification model to look after. The model uses the F1 score for comparison purposes. The challenge is that the F1 score against the test dataset is very high, like 99%, ...
DanDanDan's user avatar
  • 115
0 votes
0 answers
220 views

class_weight='balanced' vs high f_beta score for imbalanced logistic regression in sklearn. Please help explain the difference

I have an imbalanced binary classification problem I am trying to solve with the LogisticRegression algorithm in sklearn. As the data is highly imbalanced I am looking at ways to treat the imbalance ...
kdbaseball8's user avatar
0 votes
1 answer
134 views

Why True Positives equals to True Negatives while calculating micro F1 or accuracy for multi-class single label classification?

I was searching what is the difference between micro F1 and accuracy since sklearn classification_report shows accuracy in place of micro F1. I found multiple ...
Ritwik's user avatar
  • 121
0 votes
2 answers
122 views

What would be the statistically relevant patterns among $25, 18, 15, 12, 10, 8, 6, 4, 2, 1, =101{\it ?}$

Here is the current Formula One World Championship points scoring systems— Currently in Formula 1, there is a lot of talk about this and whether it is adequate or if it should be changed. $$\begin{...
user avatar
0 votes
0 answers
73 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
Andrew's user avatar
  • 13
1 vote
0 answers
89 views

Extending F1 score to classification with uncertainty

I have a classifier that can return yes, no's and "maybe"s. The maybe indicates that we don't have certainty in a prediction because the data point is is not sufficiently close to it's ...
Att Righ's user avatar
  • 141
1 vote
0 answers
622 views

How to describe f1 soft loss mathematically

Normal F1-score using binarized prediction can be described like this: $$F_1 = \frac{2 \cdot TP }{2 \cdot TP + FP + FN}$$ But in a loss function for a Machine Learning model, you will typically need ...
bjornsing's user avatar
  • 209
1 vote
1 answer
730 views

How to calculate F1, Precision, and Recall for Multi-Label Multi-Classification

I have a predictive model as follows Sample1 Sample2 Sample3 Sample4 Red Yellow Blue Green White Black Orange 65 21 55 40 0 0 1 0 1 0 0 31 40 44 30 0 0 0 0 0 0 0 33 44 56 66 1 0 0 1 0 0 1 63 77 ...
asmgx's user avatar
  • 261
1 vote
1 answer
98 views

Model accuracy versus F1

When training a model (classifier) in TensorFlow, an accuracy value is returned. What is the interpretation of an accuracy of, say, ...
Simon Lindgren's user avatar
0 votes
0 answers
151 views

How to calculate chance-level f1, ROC-AUC, PR-AUC for imbalanced dataset

I have an imbalance dataset (60% class 1, 40% class 0). I trained a model and got accuracy, f1, ROC-AUC and PR-AUC. I want to compare them to chance-level performance. obviously chance-level of acc if ...
Cranjis's user avatar
  • 57
0 votes
2 answers
430 views

What is the F1 Score for my prediction when all values are negative?

I have built a model that gives me classification of some cases here is a comparison between Actual and Prediction ...
asmgx's user avatar
  • 261
1 vote
1 answer
413 views

F1 Score is giving good value in imbalanced dataset

If I have an imbalanced dataset that consists of 90% positive points and 10% negative points. Now I created a "dumb" model which always predicts every point as a positive point. The ...
Mauj Mishra's user avatar
0 votes
0 answers
20 views

Continuously occurring true negatives

How can I handle discrete events in a continuous time stream in the context of an F1 metric? To give an example, let's say the Earthquake Forecasting Bureau would report the following for their ...
KevinB56's user avatar
2 votes
0 answers
203 views

Average Precision vs average F1

Average precision computes the area under the recall-precision curve by the trapezoidal rule (or midpoint rule). However, we could also compute the F1 score for every threshold and then take the ...
displayname's user avatar
0 votes
1 answer
35 views

hyperparameter search with unknown test set distribution

I'm training a 3-class neural network classifier (conv layers and softmax at the end, nothing special). Let's say, in the test set I will have N1 examples of the 1st class, N2 examples of the 2nd ...
sergey's user avatar
  • 51
4 votes
1 answer
266 views

Why use harmonic mean for precision and recall (f1 score) instead of just the product of precision and recall?

General question here, I understand the purpose of using the harmonic mean to generate the f1 score for model evaluation. I'm not exactly sure though why we don't just take the product of precision ...
Daniel's user avatar
  • 159
3 votes
1 answer
4k views

On which set (train/val/test) do people calculate F1 score, precision and recall?

This may be a stupid question, but when I was looking at the definition of precision/recall etc. it was not mentioned anywhere which set (training/validation/test) this metric should be calculated ...
Curaçao Hajek's user avatar
0 votes
0 answers
59 views

How to explain a relationship between Accuracy and F1 Score / F-Measure?

I am building a CNN model for pitch estimation using a song recording. For the evaluation metrics, I am using Accuracy and ...
Dionisius Pratama's user avatar
0 votes
0 answers
1k views

f1-score of imbalanced data within k fold cross validation

I am trying to find the f1 score, precision, recall of a highly imbalanced dataset. I would like to use k-fold cross validation approach. I followed the procedure: create arrays to store testing data ...
JSVJ's user avatar
  • 115
2 votes
1 answer
174 views

Harmonic is used in F1 score because it is a conservative metric: How does it help being conservative?

I was reading Jurafsky 3rd edition, page 12-13 chapter 4 Can you explain why is it good to weigh more the smaller of the two items namely $\frac{1}{Precision}$ or $\frac{1}{Recall}$? Here is the link ...
user27286's user avatar
  • 279
2 votes
1 answer
950 views

Are Jaccard score and F1 score monotonically related?

I have compared the rankings obtained by comparing 10+ classifiers with this two metrics: Jaccard score F1 score They show a perfect correlation. This results holds on 50+ datasets. When comparing ...
Adrien Pavao's user avatar
1 vote
0 answers
2k views

Are F1 score and Dice coefficient computed in same way or different way in image segmentation (two class segmentation)?

On page 8 of the paper An automatic nuclei segmentationmethod based on deep convolutional neuralnetworks for histopathology images, the authors show performance of their deep model on test sets. They ...
Prasanjit Rath's user avatar
1 vote
1 answer
803 views

F1 weighted vs. Log loss in SciKit learn RandomSearchCV

I am sorry to ask another question regarding this topic but I am still puzzled about the following: When I use 'F1_weighted' as my scoring argument in a ...
JonnDough's user avatar
  • 171
7 votes
1 answer
3k views

F1 score, PR or ROC curve for regression

Due to my background as a pure biologist, I've been struggling with the comment acquired from a reviewer about the accuracy test used in my regression study. While I stick to MSE, MAE and R2 as the ...
Tofu King's user avatar
0 votes
1 answer
48 views

Is it possible for a binary classifier to have lower accuracy, macrof1 and binaryf1 but higher ROC AUC? [duplicate]

I've got the results of two classifiers based on 5 different splits of training and testing sets. Their mean and std of the results are as follow: Method-------Accuracy -- MacroF1 -- BinaryF1---- ROC ...
user137927's user avatar