Precision and recall estimated with stratified sampling

Question

I have a population including rare events (let's call them "event A") and I want to evaluate the precision and recall of a new algorithm to detect the rare events. In the actual population, I have 100 millions events, including 10,000 A events of interest (positives). But to evaluate the new model, I cannot process the whole population, as it would take too long. So I want to use a sample that will contain all 10k A events, but only 100k negatives, drawn uniformly at random from the whole population of negatives. From the results on this sample, I would evaluate the precision (TP/(TP+FP)) and recall (TP/(TP+FN)) of the algorithm on the whole population.

My reasoning is that in the test population, the ratio of positives to negatives is 1e4/1e5 = 1/10, but in the actual population, that ratio is 1e4/1e8 = 1e-4. So, to undo the bias I introduced in the sample, I should weigh the negatives by 1/(1e-1/1e-4) = 1e3. I would still weigh the positives as 1. With those weights, I would proceed to count the TP, FP and FN in the test sample, then calculate:

$$precision = \frac{TP}{TP+10^3FP}$$ and $$recall = \frac{TP}{TP + FN}$$

The number of FPs needs to be correct, as a single FP in the test sample "stands in" for $10^3$ in the actual population. However, for the FN, a false negative is actually a positive, and those were not biased in the test sample, so I should not correct them.

Is that correct?

PS: after this post, I actually found a way to better estimate the precision, and sidestep the over-counting of FPs in the whole population by the formula above. I draw a second sample, that contains negatives only, which allows me to estimate the actual number of FPs in the population much more precisely, and I calculate the precision from the number of TPs of the first sample, and the estimated FPs for the whole population from the second sample. This procedure with 2 samples seems to evaluate the precision on the whole population much better than the formula above.

Answer 1

I'm not sure I'd call this stratified sampling, but your calculations for precision and recall make sense. It's as if you cluster negative examples into 1K groups and choose a representative among them and calculate approximated statistics based on these ones.

Of course, the effect of this procedure on the training is another issue, e.g. for bayesian approaches, your priors will be calculated differently. So, you may need to account for sample weights.

Answer 2

I think your recall equation is incorrect and you would need to scale up the FN count. You are stratifying your sample by the classifier label, and need to reweight when computing the metric.

As a toy example, imagine that your classifier had 100% precision and 50% recall, and the true population looked like this:

10k (TP)	0 (FP)
10k (FN)	99.98M (TN)

If you sample all 10k events where the classifier gave a positive label (i.e. sampling the entire first row above), you would have TP = 10k, FP = 0.

If you sample 100k examples where the classifier gave a negative label, on average you would obtain roughly 100k * (10k/99.98M) = 10.002 FN.

Using your equation, your recall would be 10k/(10k + 10.002) = 99.9%, which is clearly not correct.

In the population, the positive:negative label ratio is 1:9,999, and you sampled at a ratio of 1:10. This means that you'll need to upweight the FNs by a factor of 999.9.

Doing this would give you a recall of 10k/(10k + 10.002 * 999.9) = 50%.