$$
\text{Precision} = P(Y = 1\vert\hat Y = 1) = \dfrac{
P(\hat Y = 1\vert Y = 1)P(Y = 1)
}{
P(\hat Y = 1)
}=\dfrac{
\text{Recall}\times\text{Prevalence}
}{
P(\hat Y = 1)
}
$$
You want to calculate the prevalence and then multiply that prevalence by the number of samples in the new data.
$$
\text{Prevalence} = \dfrac{
\text{Precision}\times P(\hat Y = 1)
}{
\text{Recall}
}
$$
From your modeling, you know the precision and recall. You also know the proportion of predictions that are $1$ instead of $0$, giving you $P(\hat Y = 1)$. Therefore, you have a calculation of the prevalence that you can use as the estimation of the prevalence in the new data set. You can use that calculated prevalence to estimate the number of $Y=1$ events in the new data.
However, you should not have to go through all of that. You can just calculate the prevalence, without even developing a classifier, and use that value as the estimated prevalence in the new sample.
The simulation below gives an example where these calculations are shown to be equal.
set.seed(2024)
N <- 1000
x1 <- rnorm(N, 0, 1)
x2 <- rnorm(N, 0, 1)
z <- -1 + 2*x1 + 3*x2
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr)
L <- glm(y ~ x1 + x2, family = binomial)
phat <- predict(L, type = "response")
thresholds <- seq(-0.02, 1.02, 0.01)
for (i in 1:length(thresholds)){
yhat <- ifelse(phat > thresholds[i], 1, 0)
model_precision <- mean(y[yhat == 1])
model_recall <- mean(yhat[y == 1])
py1 <- mean(yhat)
prevalence <- mean(y)
# Print the difference in the directly calculated prevalence vs
# the prevalence calculated from Bayes' theorem on the precision
#
print((model_precision*py1/model_recall) - prevalence)
}
# I get a difference of basically zero at every threshold for which precision exists.
Here is an example with a validation set.
set.seed(2024)
set.seed(2024)
N <- 1000
x1 <- rnorm(N, 0, 1)
x2 <- rnorm(N, 0, 1)
z <- -1 + 2*x1 + 3*x2
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr)
L <- glm(y ~ x1 + x2, family = binomial)
x1_val <- rnorm(N, 0, 1)
x2_val <- rnorm(N, 0, 1)
z_val <- -1 + 2*x1_val + 3*x2_val
pr_val <- 1/(1 + exp(-z_val))
y_val <- rbinom(N, 1, pr_val)
d_val <- data.frame(x1 = x1_val, x2 = x2_val)
phat <- predict(L, type = "response", data = d_val)
thresholds <- seq(-0.02, 1.02, 0.01)
for (i in 1:length(thresholds)){
yhat <- ifelse(phat > thresholds[i], 1, 0)
model_precision <- mean(y_val[yhat == 1])
model_recall <- mean(yhat[y_val == 1])
py1 <- mean(yhat)
prevalence <- mean(y_val)
# Print the difference in the directly calculated prevalence vs
# the prevalence calculated from Bayes' theorem on the precision
#
print((model_precision*py1/model_recall) - prevalence)
}
# I get a difference of basically zero at every threshold for which precision exists.
