Skip to main content
fixed a small mistake
Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

The definitions of precision and recall are: $$ \begin{align} precision &= \frac{TP}{TP+FP}, \\ recall &= \frac{TP}{TP+FN}, \end{align} $$ with TP, FP and FN the number of true positives, false positives and false negatives, respectively.

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$$$ \begin{align} precision &= [0, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking ($\frac{4}{6})$. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at cutoffs that are higher than the first cutoffhighest we observed (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

The definitions of precision and recall are: $$ \begin{align} precision &= \frac{TP}{TP+FP}, \\ recall &= \frac{TP}{TP+FN}, \end{align} $$ with TP, FP and FN the number of true positives, false positives and false negatives, respectively.

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking ($\frac{4}{6})$. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at the first cutoff (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

The definitions of precision and recall are: $$ \begin{align} precision &= \frac{TP}{TP+FP}, \\ recall &= \frac{TP}{TP+FN}, \end{align} $$ with TP, FP and FN the number of true positives, false positives and false negatives, respectively.

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [0, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking ($\frac{4}{6})$. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at cutoffs that are higher than the highest we observed (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

added 16 characters in body
Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

The definitions of precision and recall are: $$ \begin{align} precision &= \frac{TP}{TP+FP}, \\ recall &= \frac{TP}{TP+FN}, \end{align} $$ with TP, FP and FN the number of true positives, false positives and false negatives, respectively.

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking ($\frac{4}{6})$. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at the first cutoff (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at the first cutoff (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

The definitions of precision and recall are: $$ \begin{align} precision &= \frac{TP}{TP+FP}, \\ recall &= \frac{TP}{TP+FN}, \end{align} $$ with TP, FP and FN the number of true positives, false positives and false negatives, respectively.

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking ($\frac{4}{6})$. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at the first cutoff (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.

Source Link
Marc Claesen
  • 18.7k
  • 2
  • 55
  • 76

This may be counterintuitive, but precision is not necessarily monotonically decreasing in terms of the classification threshold. On the other hand, recall is monotonically increasing. (I am assuming you rank data in terms of decreasing classifier scores, which appears opposite to what your example does, but does not change the conclusion)

Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:

$$[False, True, False, True, True, True, False, False],$$

which leads to the following points in precision-recall space:

$$ \begin{align} precision &= [\star, \frac{1}{2}, \frac{1}{3}, \frac{2}{4}, \frac{3}{5}, \frac{4}{6}, \frac{4}{7}, \frac{4}{8}], \\ recall &= [0, \frac{1}{4}, \frac{1}{4}, \frac{2}{4}, \frac{3}{4}, \frac{4}{4}, \frac{4}{4}, \frac{4}{4}]. \end{align} $$

As you can see, recall is monotonically increasing but precision has a maximum somewhere in the middle of the ranking. The shape of precision in terms of threshold can take any form, but usually you will have high precision at high thresholds and vice versa.

Note that precision is undefined at the first cutoff (0/0), but usually we say this is 0, for example when computing area under the precision-recall curve.