# Is it possible to estimate the number of positives from precision and recall values?

Let's say, I have a binary predictor, and its performance in precision and recall is known from the previous study. Now, we apply the predictor on the new (unknown) dataset with 1000 samples, and got 80 predicted positives.

From our previous study, we know that precision = 0.99 and recall = 0.6.

Since precision = TP/PP and recall = TP/P. In order to calculate the total number of positive samples (P), I can calculate:

80 * precision / recall = 132

Obviously I have to assume that the predictor works exactly the same as the one in the new dataset.

Is the above argument sound? Am I missing something here?

• If you can calculate the precision and recall, you already know how many positives there are. (You must know which observations are positive and negative to calculate precision and recall.) Why not just count them?
– Dave
Commented Apr 17 at 21:31
• I am talking about the new data set that we don't have ground truth. Commented Apr 18 at 22:18
• Then how can you have the precision and recall values?
– Dave
Commented Apr 19 at 0:28
• because I calculate precision and recall values from the validation set where I have the ground truth? Commented Apr 29 at 21:19
• Then why can’t you just count the number of positives?
– Dave
Commented Apr 29 at 22:04

$$\text{Precision} = P(Y = 1\vert\hat Y = 1) = \dfrac{ P(\hat Y = 1\vert Y = 1)P(Y = 1) }{ P(\hat Y = 1) }=\dfrac{ \text{Recall}\times\text{Prevalence} }{ P(\hat Y = 1) }$$

You want to calculate the prevalence and then multiply that prevalence by the number of samples in the new data.

$$\text{Prevalence} = \dfrac{ \text{Precision}\times P(\hat Y = 1) }{ \text{Recall} }$$

From your modeling, you know the precision and recall. You also know the proportion of predictions that are $$1$$ instead of $$0$$, giving you $$P(\hat Y = 1)$$. Therefore, you have a calculation of the prevalence that you can use as the estimation of the prevalence in the new data set. You can use that calculated prevalence to estimate the number of $$Y=1$$ events in the new data.

However, you should not have to go through all of that. You can just calculate the prevalence, without even developing a classifier, and use that value as the estimated prevalence in the new sample.

The simulation below gives an example where these calculations are shown to be equal.

set.seed(2024)
N <- 1000
x1 <- rnorm(N, 0, 1)
x2 <- rnorm(N, 0, 1)
z <- -1 + 2*x1 + 3*x2
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr)
L <- glm(y ~ x1 + x2, family = binomial)
phat <- predict(L, type = "response")
thresholds <- seq(-0.02, 1.02, 0.01)

for (i in 1:length(thresholds)){
yhat <- ifelse(phat > thresholds[i], 1, 0)
model_precision <- mean(y[yhat == 1])
model_recall    <- mean(yhat[y == 1])
py1 <- mean(yhat)
prevalence <- mean(y)

# Print the difference in the directly calculated prevalence vs
# the prevalence calculated from Bayes' theorem on the precision
#
print((model_precision*py1/model_recall) - prevalence)
}

# I get a difference of basically zero at every threshold for which precision exists.


Here is an example with a validation set.

set.seed(2024)
set.seed(2024)
N <- 1000
x1 <- rnorm(N, 0, 1)
x2 <- rnorm(N, 0, 1)
z <- -1 + 2*x1 + 3*x2
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr)
L <- glm(y ~ x1 + x2, family = binomial)
x1_val <- rnorm(N, 0, 1)
x2_val <- rnorm(N, 0, 1)
z_val <- -1 + 2*x1_val + 3*x2_val
pr_val <- 1/(1 + exp(-z_val))
y_val <- rbinom(N, 1, pr_val)
d_val <- data.frame(x1 = x1_val, x2 = x2_val)
phat <- predict(L, type = "response", data = d_val)
thresholds <- seq(-0.02, 1.02, 0.01)

for (i in 1:length(thresholds)){
yhat <- ifelse(phat > thresholds[i], 1, 0)
model_precision <- mean(y_val[yhat == 1])
model_recall    <- mean(yhat[y_val == 1])
py1 <- mean(yhat)
prevalence <- mean(y_val)

# Print the difference in the directly calculated prevalence vs
# the prevalence calculated from Bayes' theorem on the precision
#
print((model_precision*py1/model_recall) - prevalence)
}

# I get a difference of basically zero at every threshold for which precision exists.