Classification paradox: number of elements in each class from recall and precision

Question

Suppose that we have

• k classes, $C_1$,...$C_k$.
• A classifier with precision $p_i$ and recall $c_i$, for $i=1 \dots k$.
• A sample S, such that the classifier assigns $s_i$ elements of S to class $C_i$ fo $i=1 \dots k$.

I'm not interested in which particular elements correspond to each class, but in the true number of elements that correspond to each class $r_i$.

That is, I'm interested in obtaining $r_i$ (the real number of elements in class $C_i$) from $s_i$ (the number of elements classified as $C_i$), the precision $p_i$ and the recall $r_i$.

An obvious answer would be

$$ (1) \qquad r_i = s_ip_i + (1-r_i)s_ip_i $$ that is, of the $s_i$ elements classified as $C_i$, only $s_ip_i$ are indeed in $C_i$. This number must be incremented taking into account the recall, or more precisely $(1-r_i)$, which gives the second value $(1-r_i)s_ip_i$.

I would like to know if $(1)$ is ok. It sounds strange to me, because apparentely it allows obtaining the value $r_i$ even if the precision $p_i$ and recall $r_i$ are very low, that is, we can obtain the real number of elements even if the classification method is really poor.

What is the answer to this paradox?

Answer 1

An obvious answer would be $$ (1) \qquad r_i = s_ip_i + (1-r_i)s_ip_i $$

To begin with, it looks like you're reusing $r_i$ here for both the recall and the class size. Assuming the class size is $c_i$, and the precision and recall are $p_i$ and $r_i$, you probably meant

\begin{equation} c_i = s_i p_i + (1 - r_i) s_i p_i. \end{equation}

Assuming that's the case, I disagree with the equation. Consider the following diagram taken from the relevant Wikipedia entry

A reasonable approximation of the number of green elements is indeed $s_i p_i$. However, following that, an approximation of the number of dark-grey (including green) elements, would be

\begin{equation} c_i \sim \frac{s_i p_i}{r_i}. (2) \end{equation}

For example, suppose the classifier has $p_i = 1, r_i = \epsilon \ll 1$, which means that the red part is empty, and the green part is a tiny fraction of the rest of the dark-grey part. Using your (1), you would get $c_i \sim s_i$, which seems incorrect. Using (2), though $c_i \gg s_i$.

It sounds strange to me, because apparentely it allows obtaining the value ri even if the precision pi and recall ri are very low, that is, we can obtain the real number of elements even if the classification method is really poor.

If $r_i$ is low, the estimation of $c_i$ is unstable (in the extreme case, if $r_i = 0$, it is undefined).

If $r_i$ is high, though, then the green part is a large of the dark-grey part (in the extreme $r_i = 1$, and the dark-grey part is a subset of the green part). If this is the case, then $p_i s_i$ is a good approximation.