Why f beta score define beta like that?

Question

This is the F beta score: $$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$

The Wikipedia article states that $F_\beta$ "measures the effectiveness of retrieval with respect to a user who attaches β times as much importance to recall as precision".

I did not get the idea. Why define $\beta$ like that? Can I define $F_\beta$ like this:

$$F_\beta = (1 + \beta) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta \cdot \mathrm{precision}) + \mathrm{recall}}$$

And how to show β times as much importance?

Answer 1

Letting $\beta$ be the weight in the first definition you provide and $\tilde\beta$ the weight in the second, the two definitions are equivalent when you set $\tilde\beta = \beta^2$, so these two definitions represent only notational differences in the definition of the $F_\beta$ score. I have seen it defined both the first way (e.g. on the wikipedia page) and the second (e.g. here).

The $F_1$ measure is obtained by taking the harmonic mean of precision and recall, namely the reciprocal of the average of the reciprocal of precision and the reciprocal of recall:

\begin{align*} F_1 &= \frac{1}{\frac{1}{2}\frac{1}{\text{precision}}+\frac{1}{2}\frac{1}{\text{recall}}} \\ &= 2\frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} \end{align*}

Instead of using weights in the denominator that are equal and sum to 1 ($\frac{1}{2}$ for recall and $\frac{1}{2}$ for precision), we might instead assign weights that still sum to 1 but for which the weight on recall is $\beta$ times as large as the weight on precision ($\frac{\beta}{\beta+1}$ for recall and $\frac{1}{\beta+1}$ for precision). This yields your second definition of the $F_\beta$ score:

\begin{align*} F_\beta &= \frac{1}{\frac{1}{\beta+1}\frac{1}{\text{precision}}+\frac{\beta}{\beta+1}\frac{1}{\text{recall}}} \\ &= (1+\beta)\frac{\text{precision}\cdot\text{recall}}{\beta\cdot\text{precision}+\text{recall}} \end{align*}

Again, if we had used $\beta^2$ instead of $\beta$ here we would have arrived at your first definition, so the differences between the two definitions are just notational.

Answer 2

The reason for defining the F-beta score with $\beta^{2}$ is exactly the quote you provide (i.e. wanting to attach $\beta$ times as much importance to recall as precision) given a particular definition for what it means to attach $\beta$ times as much importance to recall than precision.

The particular way of defining the relative importance of the two metrics that leads to the $\beta^{2}$ formulation can be found in Information Retrieval (Van Rijsbergen, 1979):

Definition: The relative importance a user attaches to precision and recall is the $P/R$ ratio at which $\partial{E}/ \partial{R} = \partial{E}/ \partial{P}$, where $E = E(P, R)$ is the measure of effectiveness based on precision and recall.

The motivation for this being:

The simplest way I know of quantifying this is to specify the $P/R$ ratio at which the user is willing to trade an increment in precision for an equal loss in recall.

To see that this leads to the $\beta^{2}$ formulation we can start with the general formula for the weighted harmonic mean of $P$ and $R$ and calculate their partial derivatives with respect to $P$ and $R$. The source cited uses $E$ (for "effectiveness measure"), which is just $1-F$ and the explanation is equivalent whether we consider $E$ or $F$.

\begin{equation} F = \frac{1}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})} \end{equation}

\begin{equation} \partial{F}/\partial{P} = \frac{\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}P^{2}} \end{equation}

\begin{equation} \partial{F}/\partial{R} = \frac{1-\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}R^{2}} \end{equation}

Now, setting the derivatives equal to one another places a restriction on the relationship between $\alpha$ and the ratio $P/R$. Given that we wish to attach $\beta$ times as much importance to recall as precision we will consider the ratio $R/P$¹:

\begin{equation} \partial{F}/\partial{P} = \partial{F}/\partial{R} \rightarrow \frac{\alpha}{P^{2}} = \frac{1-\alpha}{R^{2}} \rightarrow \frac{R}{P} = \sqrt{\frac{1-\alpha}{\alpha}} \end{equation}

Defining $\beta$ as this ratio and rearranging for $\alpha$ gives the weightings in terms of $\beta^{2}$:

\begin{equation} \beta = \sqrt{\frac{1-\alpha}{\alpha}} \rightarrow \beta^{2} = \frac{1-\alpha}{\alpha} \rightarrow \beta^{2} + 1 = \frac{1}{\alpha} \rightarrow \alpha = \frac{1}{\beta^{2} + 1} \end{equation}

\begin{equation} 1 - \alpha = 1 - \frac{1}{\beta^{2} + 1} \rightarrow \frac{\beta^{2}}{\beta^{2} + 1} \end{equation}

We obtain:

\begin{equation} F = \frac{1}{(\frac{1}{\beta^{2} + 1}\frac{1}{P} + \frac{\beta^{2}}{\beta^{2} + 1}\frac{1}{R})} \end{equation}

Which can be rearranged to give the form in your question.

Thus, given the quoted definition, if you wish to attach $\beta$ times as much importance to recall as precision then the $\beta^{2}$ formulation should be used. This interpretation does not hold if one uses $\beta$.

You could define a score as you suggest. In this case, as Vic has shown, the definition for the relative importance you would be assuming is:

Definition: The relative importance a user attaches to precision and recall is the $(\partial{E}/ \partial{R}) / (\partial{E}/ \partial{P})$ ratio at which $R = P$.

Footnotes:

1. $P/R$ is used in Information Retrieval but this appears to be a typo, see The Truth of F-measure (Saski, 2007).

References:

1. C. J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.), pp.133-134
2. Y. Sasaki. 2007. “The Truth of F-measure”, Teaching, Tutorial materials

Answer 3

To point something out quickly.

It means that as the beta value increases, you value precision more.

I actually think it's the opposite - since higher is better in F-β scoring, you want the denominator to be small. Therefore, if you decrease β, then the model is punished less for having a good precision score. If you increase β, then the F-β score is punished more when precision is high.

If you want to weight the F-β scoring so that it values precision, β should be 0 < β < 1, where β->0 values only precision (the numerator becomes very small, and the only thing in the denominator is recall, so the F-β score decreases as recall increases).

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html

Answer 4

TLDR; Contrary to the literature which all traces back to an arbitrary proposed definition, using a $\beta$ term like OP suggests is actually more intuitive than the $\beta^2$ term.

A Person's answer does well to show why $\beta^{2}$ appears, given Van Rijsbergen's chosen way to define the relative importance of precision and recall. However, there is a consideration that's missing in the literature, which I'm arguing here: the chosen definition is unintuitive and unnatural, and if you actually used $F_\beta$ (in practice) the way it's defined, you would quickly be left thinking, "the effect of $\beta$ seems way more aggressive than the value I've chosen".

To be fair, it is mostly Wikipedia's summary that is misleading, as it neglects to mention the subjective measure of importance involved, whereas Van Rijsbergen merely presented a possible definition that was simple but not necessarily the best or most meaningful one.

Let's review Van Rijsbergen's choice of definition:

The simplest way I know of quantifying this is to specify the $P/R$ ratio at which the user is willing to trade an increment in precision for an equal loss in recall.

Generally speaking, if $R/P > \beta$ then an increase in $P$ is more influential than an increase in $R$, whereas $R$ is more influential than $P$ where $R/P < \beta$. But here's why I would argue that the weighting is unintuitive. When $P = R$, increases in $R$ are $\beta^2$ times as effective as $P$. (This can be calculated from the partial derivatives provided in A Person's answer.) When someone says "I want recall to be weighted 3x more important than precision", I would not jump to the definition that equates to "precision will be penalised until it's literally a third of the value of recall", and I certainly wouldn't expect that when precision and recall are equal, recall contributes 9x as much. That doesn't seem practical in most situations where you ideally want both precision and recall to be high, just one to be a little higher than the other.

Below is a visual representation of what $F_\beta$ looks like. The red lines highlight the ratio $R/P = \beta$ and that the partial derivatives of $F_\beta$ are equal at that ratio, shown by the solid red slopes.

I'll now present an alternative subjective definition, which equates to "when precision and recall are equal, improvements in recall are worth $\gamma$ times more than improvements in precision". I argue that this definition is more intuitive while being equally simple as Van Rijsbergen's definition:

When $P = R$, set $\frac{\partial{F}/\partial{R}}{\partial{F}/\partial{P}} = \gamma$, where $\gamma$ is the relative importance of improvements in recall over precision.

Substituting equations derived in A Person's answer:

$\frac{1-\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}R^{2}} = \gamma \frac{\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}P^{2}}$

Remembering that $P = R$, this simplifies to:

$\gamma = \frac{1-\alpha}{\alpha}$ and $\alpha = \frac{1}{\gamma + 1}$,

contrasted with:

$\beta^2 = \frac{1-\alpha}{\alpha}$ and $\alpha = \frac{1}{\beta^2+1}$ under Van Rijsbergen's formulation.

What does this mean? An informal summary:

• Van Rijsbergen's definition $\Leftrightarrow$ recall is $\beta$ times as important as precision in terms of value.
• My proposed definition $\Leftrightarrow$ recall is $\gamma$ times as important as precision in terms of improvements in value.
• Both definitions are based on a weighted harmonic mean of precision and recall, and the weightings under these two definitions can be mapped. Specifically, placing $\beta = \sqrt{\gamma}$ times importance in terms of value is equivalent to placing $\gamma$ times importance in terms of improvements in value.
• One can defensibly argue that using a $\beta$ term instead of $\beta^2$ is a more intuitive weighting.

Answer 5

The reason that β^2 is multiplied with precision is just the way that F-Scores are defined. It means that as the beta value increases, you value precision more. If you wanted to multiply it with recall that would also work, it would just mean that as the beta value increases you value recall more.

Answer 6

The beta value greater than 1 means we want our model to pay more attention to the model Recall as compared to Precision. On the other, a value of less than 1 puts more emphasis on Precision.