2
$\begingroup$

We can compute the F-1 score in the following two ways.

  1. $F_{1_{PRE, REC}} = 2 * (PRE * REC) / (PRE + REC)$

  2. $F_{1_{TP, FP, FN}} = (2 * TP) / (2 * TP + FP + FN)$

Both computes F1 score, but which one is more appropriate to use in practice to classify highly imbalanced data with binary response? I believe Sklearn python library uses the first one.Thanks!

$\endgroup$
1

1 Answer 1

4
$\begingroup$

The formulas are exactly equivalent, as you can see from the Wikipedia page. So if you want to calculate an F1 score, it doesn't matter which you choose as you get the same result.

As to being "appropriate," many who frequent this site would argue that the F1 score isn't "appropriate" in most circumstances. It is based on an arbitrary choice of a probability cutoff for making the class assignments (typically a hidden assumption of p = 0.5 for the cutoff), and it implicitly assumes that recall and precision are equally important. To deal with the latter issue, you at least should calculate the $F_{\beta}$ score when "recall is considered $\beta$ times as important as precision" (Wikipedia):

$$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}} $$

Furthermore, all the F scores ignore the true negatives. Thus if you switch the choice of the positive class you get a different F score. Implications of that problem and suggestions for better model-assessment approaches are on this page and on many others linked from a comment on the present question. Build a good probability model based on a proper scoring rule, and only then choose a probability cutoff based on the relative costs of false-positive and false-negative decision. If you then want to report an F score, fine, but don't use it for building and assessing the model.

$\endgroup$
6
  • $\begingroup$ I see most articles in the literature recommended to use F1 score or G-mean to evaluate the model. $\endgroup$ Mar 20, 2021 at 21:59
  • 2
    $\begingroup$ I personally would say that most articles are simply wrong. Yes, I really do believe that I know better than the majority of the literature. I would very much recommend you take a look at the linked threads, Tilmann Gneiting's papers on proper scoring rules (see the tag wiki) and the arguments presented by Frank Harrell. ... $\endgroup$ Mar 21, 2021 at 6:11
  • 2
    $\begingroup$ ... You will notice a pattern: people who recommend the F1 score or similar KPIs (like accuracy, sensitivity, specificity etc.) are invariably non-statisticians, e.g., data scientists who come from computer science, while the statistical experts will recommend proper scoring rules. Nothing against computer science, but the "accuracy" camp simply has never engaged with the probabilistic nature of classification. (One exception: weather forecasters have already since the 1950s assessed their forecasts using proper scoring rules, see Brier.) $\endgroup$ Mar 21, 2021 at 6:13
  • $\begingroup$ @StephanKolassa Could u please tell what would be proper scoring rule? I see people has been using either g-mean or F1 score to evaluate the binary classification of imbalanced data! Do u recommend Brier score? If so, could u please provide any article to justify the use of Brier score to evaluate imbalanced data? I appreciate your time! $\endgroup$ Apr 3, 2021 at 4:24
  • $\begingroup$ @Simpson'sParadox: If you use any proper scoring rule, the "problem" with unbalanced data completely disappears. See this thread. Much more information can be found in the tag wiki. As to which scoring rule to use, this thread compares the Brier and the log score. $\endgroup$ Apr 3, 2021 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.