3
$\begingroup$

There's some discussion on what F-measure means. I understand that the beta parameter determines the weight of recall in the combined score. In specific one answer states that "for good models using the $F_{\beta}$ implies you consider false negatives $\beta^2$ times more costly than false positives." beta < 1 lends more weight to precision, while beta > 1 favors recall (beta -> 0 considers only precision, beta -> +inf only recall).

If you want to weight precision or recall higher than the other, how do you decide on the beta? I'm a bit unclear on the math behind the F-measure, so does a beta = .5 mean that precision is weighted 2x as much as recall?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ From $\beta^2$, $\beta=0.5$ would suggest that precision would be weighted 4 times as much as recall, at least according to the one answer cited. $\endgroup$
    – Carl
    Feb 29, 2020 at 2:14

1 Answer 1

Reset to default
4
$\begingroup$

Don't use F scores at all. Every criticism of accuracy collected at Why is accuracy not the best measure for assessing classification models? applies completely equally to precision, recall and all F scores. Instead, use proper scoring rules.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I have a sneaking suspicion that there's additional concern with F-scores because they combine conditional probabilities with very different condition: Would you care to have a look at the last part of stats.stackexchange.com/a/99921/4598 and let me know what you think of that? $\endgroup$
    – cbeleites unhappy with SX
    2 days ago
  • $\begingroup$ I completely share your concerns about proper scoring rules, but it is possible to derive quadratic errors analogously to Brier's score that capture certain aspects of predictive behaviour like sensitivity, specificity or the predictive values (and in consequence, also an analogoue to the F-measures). They have the continuous behaviour and favorable variance properties of Brier's score/MSE, but are not themselves proper scoring rules since they focus on subgroups of cases. They can be combined, though (e.g. into Brier's score). Paper: arxiv.org/abs/1301.0264 $\endgroup$
    – cbeleites unhappy with SX
    2 days ago
  • $\begingroup$ ... I thus feel like the improper scoring aspect could be repaired. One may argue that they belong more to the decision aspect than the probability density prediction. Still I think it may be sensible to (even outside the application scenario we discuss in the paper) to have metrics at hand for the decision stage that use a closely related metric to what was used before. $\endgroup$
    – cbeleites unhappy with SX
    2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.