10
$\begingroup$

This is the F beta score: $$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$$

The Wikipedia article states that $F_\beta$ "measures the effectiveness of retrieval with respect to a user who attaches β times as much importance to recall as precision".

I did not get the idea. Why define $\beta$ like that? Can I define $F_\beta$ like this:

$$F_\beta = (1 + \beta) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta \cdot \mathrm{precision}) + \mathrm{recall}}$$

And how to show β times as much importance?

$\endgroup$
  • 2
    $\begingroup$ Check out a newer answer below that includes the differential calculus that addresses "why Beta squared and not Beta". $\endgroup$ – javadba Dec 10 '18 at 7:24
17
$\begingroup$

Letting $\beta$ be the weight in the first definition you provide and $\tilde\beta$ the weight in the second, the two definitions are equivalent when you set $\tilde\beta = \beta^2$, so these two definitions represent only notational differences in the definition of the $F_\beta$ score. I have seen it defined both the first way (e.g. on the wikipedia page) and the second (e.g. here).

The $F_1$ measure is obtained by taking the harmonic mean of precision and recall, namely the reciprocal of the average of the reciprocal of precision and the reciprocal of recall:

\begin{align*} F_1 &= \frac{1}{\frac{1}{2}\frac{1}{\text{precision}}+\frac{1}{2}\frac{1}{\text{recall}}} \\ &= 2\frac{\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} \end{align*}

Instead of using weights in the denominator that are equal and sum to 1 ($\frac{1}{2}$ for recall and $\frac{1}{2}$ for precision), we might instead assign weights that still sum to 1 but for which the weight on recall is $\beta$ times as large as the weight on precision ($\frac{\beta}{\beta+1}$ for recall and $\frac{1}{\beta+1}$ for precision). This yields your second definition of the $F_\beta$ score:

\begin{align*} F_\beta &= \frac{1}{\frac{1}{\beta+1}\frac{1}{\text{precision}}+\frac{\beta}{\beta+1}\frac{1}{\text{recall}}} \\ &= (1+\beta)\frac{\text{precision}\cdot\text{recall}}{\beta\cdot\text{precision}+\text{recall}} \end{align*}

Again, if we had used $\beta^2$ instead of $\beta$ here we would have arrived at your first definition, so the differences between the two definitions are just notational.

$\endgroup$
  • 1
    $\begingroup$ why did they multiply $\beta$ with the precision term instead of the recall term? $\endgroup$ – Anwarvic Jun 7 '18 at 12:07
  • 1
    $\begingroup$ The differential calculus that addresses "why Beta squared and not Beta" is included in a newer answer below. $\endgroup$ – javadba Dec 10 '18 at 7:21
  • $\begingroup$ @Anwarvic They multiplied $\beta$ with the inverse recall. After factoring out $(1+ \beta)$ and expanding with $\text{precision} \cdot \text{recall}$ there is a $\beta \cdot \text{precision}$ term left $\endgroup$ – user2740 Aug 13 at 9:10
5
$\begingroup$

The reason for defining the F-beta score with $\beta^{2}$ is exactly the quote you provide (i.e. wanting to attach $\beta$ times as much importance to recall as precision) given a particular definition for what it means to attach $\beta$ times as much importance to recall than precision.

The particular way of defining the relative importance of the two metrics that leads to the $\beta^{2}$ formulation can be found in Information Retrieval (Van Rijsbergen, 1979):

Definition: The relative importance a user attaches to precision and recall is the $P/R$ ratio at which $\partial{E}/ \partial{R} = \partial{E}/ \partial{P}$, where $E = E(P, R)$ is the measure of effectiveness based on precision and recall.

The motivation for this being:

The simplest way I know of quantifying this is to specify the $P/R$ ratio at which the user is willing to trade an increment in precision for an equal loss in recall.

To see that this leads to the $\beta^{2}$ formulation we can start with the general formula for the weighted harmonic mean of $P$ and $R$ and calculate their partial derivatives with respect to $P$ and $R$. The source cited uses $E$ (for "effectiveness measure"), which is just $1-F$ and the explanation is equivalent whether we consider $E$ or $F$.

\begin{equation} F = \frac{1}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})} \end{equation}

\begin{equation} \partial{F}/\partial{P} = \frac{\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}P^{2}} \end{equation}

\begin{equation} \partial{F}/\partial{R} = \frac{1-\alpha}{(\frac{\alpha}{P}+ \frac{1-\alpha}{R})^{2}R^{2}} \end{equation}

Now, setting the derivatives equal to one another places a restriction on the relationship between $\alpha$ and the ratio $P/R$. Given that we wish to attach $\beta$ times as much importance to recall as precision we will consider the ratio $R/P$1:

\begin{equation} \partial{F}/\partial{P} = \partial{F}/\partial{R} \rightarrow \frac{\alpha}{P^{2}} = \frac{1-\alpha}{R^{2}} \rightarrow \frac{R}{P} = \sqrt{\frac{1-\alpha}{\alpha}} \end{equation}

Defining $\beta$ as this ratio and rearranging for $\alpha$ gives the weightings in terms of $\beta^{2}$:

\begin{equation} \beta = \sqrt{\frac{1-\alpha}{\alpha}} \rightarrow \beta^{2} = \frac{1-\alpha}{\alpha} \rightarrow \beta^{2} + 1 = \frac{1}{\alpha} \rightarrow \alpha = \frac{1}{\beta^{2} + 1} \end{equation}

\begin{equation} 1 - \alpha = 1 - \frac{1}{\beta^{2} + 1} \rightarrow \frac{\beta^{2}}{\beta^{2} + 1} \end{equation}

We obtain:

\begin{equation} F = \frac{1}{(\frac{1}{\beta^{2} + 1}\frac{1}{P} + \frac{\beta^{2}}{\beta^{2} + 1}\frac{1}{R})} \end{equation}

Which can be rearranged to give the form in your question.

Thus, given the quoted definition, if you wish to attach $\beta$ times as much importance to recall as precision then the $\beta^{2}$ formulation should be used. This interpretation does not hold if one uses $\beta$. The equivalent, less intuitive, interpretation in the case that we just use $\beta$ would be that we want to attach $\sqrt{\beta}$ times as much importance to recall as precision.

You could define a score as you suggest, however you should be aware that in this case either the interpretation discussed no longer holds or you are implying some other definition for quantifying the trade off between precision and recall.

Footnotes:

  1. $P/R$ is used in Information Retrieval but this appears to be a typo, see The Truth of F-measure (Saski, 2007).

References:

  1. C. J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.), pp.133-134
  2. Y. Sasaki. 2007. “The Truth of F-measure”, Teaching, Tutorial materials
$\endgroup$
  • 1
    $\begingroup$ This should be the accepted answer. $\endgroup$ – javadba Dec 10 '18 at 7:20
3
$\begingroup$

To point something out quickly.

It means that as the beta value increases, you value precision more.

I actually think it's the opposite - since higher is better in F-β scoring, you want the denominator to be small. Therefore, if you decrease β, then the model is punished less for having a good precision score. If you increase β, then the F-β score is punished more when precision is high.

If you want to weight the F-β scoring so that it values precision, β should be 0 < β < 1, where β->0 values only precision (the numerator becomes very small, and the only thing in the denominator is recall, so the F-β score decreases as recall increases).

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html

$\endgroup$
0
$\begingroup$

The reason that β^2 is multiplied with precision is just the way that F-Scores are defined. It means that as the beta value increases, you value precision more. If you wanted to multiply it with recall that would also work, it would just mean that as the beta value increases you value recall more.

$\endgroup$
0
$\begingroup$

The beta value greater than 1 means we want our model to pay more attention to the model Recall as compared to Precision. On the other, a value of less than 1 puts more emphasis on Precision.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.