7
$\begingroup$

I have a population including rare events (let's call them "event A") and I want to evaluate the precision and recall of a new algorithm to detect the rare events. In the actual population, I have 100 millions events, including 10,000 A events of interest (positives). But to evaluate the new model, I cannot process the whole population, as it would take too long. So I want to use a sample that will contain all 10k A events, but only 100k negatives, drawn uniformly at random from the whole population of negatives. From the results on this sample, I would evaluate the precision (TP/(TP+FP)) and recall (TP/(TP+FN)) of the algorithm on the whole population.

My reasoning is that in the test population, the ratio of positives to negatives is 1e4/1e5 = 1/10, but in the actual population, that ratio is 1e4/1e8 = 1e-4. So, to undo the bias I introduced in the sample, I should weigh the negatives by 1/(1e-1/1e-4) = 1e3. I would still weigh the positives as 1. With those weights, I would proceed to count the TP, FP and FN in the test sample, then calculate:

$$precision = \frac{TP}{TP+10^3FP}$$ and $$recall = \frac{TP}{TP + FN}$$

The number of FPs needs to be correct, as a single FP in the test sample "stands in" for $10^3$ in the actual population. However, for the FN, a false negative is actually a positive, and those were not biased in the test sample, so I should not correct them.

Is that correct?

PS: after this post, I actually found a way to better estimate the precision, and sidestep the over-counting of FPs in the whole population by the formula above. I draw a second sample, that contains negatives only, which allows me to estimate the actual number of FPs in the population much more precisely, and I calculate the precision from the number of TPs of the first sample, and the estimated FPs for the whole population from the second sample. This procedure with 2 samples seems to evaluate the precision on the whole population much better than the formula above.

$\endgroup$
6
  • 1
    $\begingroup$ Don't use accuracy, precision, recall, sensitivity, specificity, or the F1 score. Every criticism at the following threads applies equally to all of these, and indeed to all evaluation metrics that rely on hard classifications: Why is accuracy not the best measure for assessing classification models? Unbalanced classes are almost certainly not a problem, and oversampling will not solve a non-problem: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$ Aug 29, 2021 at 8:53
  • $\begingroup$ @StephanKolassa - that's all good - what kind of evaluation should one use on that type of problem then? $\endgroup$
    – Frank
    Aug 29, 2021 at 15:16
  • $\begingroup$ Sorry for only getting back now, I was on vacation. I would strongly recommend looking at probabilistic predictions, and assessing these using proper scoring rules. Take a look at the two threads I linked to in my previous comment, as well as at Classification probability threshold. $\endgroup$ Sep 7, 2021 at 15:06
  • $\begingroup$ I ended up settling on "precision @k", which means, take e.g. the top 100 model scores, and compute the precision/recall based on that only. The premise is that when it comes to putting a system in production, it would be a valid thing to do in this case. Needless to say, the precision/recall become very good $\endgroup$
    – Frank
    Sep 8, 2021 at 16:48
  • $\begingroup$ You say the ratio of positives to negatives in the population is 1e4/1e8 = 1e-4. But shouldn't it be 1e4/(1e8 - 1e4) = 1e4/(9999e4) = 1/9999? $\endgroup$ Sep 30, 2022 at 16:33

2 Answers 2

4
$\begingroup$

I'm not sure I'd call this stratified sampling, but your calculations for precision and recall make sense. It's as if you cluster negative examples into 1K groups and choose a representative among them and calculate approximated statistics based on these ones.

Of course, the effect of this procedure on the training is another issue, e.g. for bayesian approaches, your priors will be calculated differently. So, you may need to account for sample weights.

$\endgroup$
11
  • 1
    $\begingroup$ Agree -- except that I would call this stratified sampling. Actually, I'd probably call it case-control sampling, but that's a special case of stratified sampling $\endgroup$ Aug 28, 2021 at 23:23
  • $\begingroup$ That's really interesting and good to know, a stratified subsample would contain only 10 positives in the def. I'm accustomed to (splitting the larger pop into homogeneous smaller samples). $\endgroup$
    – gunes
    Aug 28, 2021 at 23:30
  • $\begingroup$ It's surprising to me that this way of drawing the sample is going to affect the precision calculation, but not the recall - is that correct? - and yes, I think this is stratified sampling, as I'm working with a partition of the original population - the only catch is that I don't sample one of the subset, because it's small enough, so I take it all, and only the one that's too large is sampled. But I should be allowed to do that and still call that "stratified sampling". $\endgroup$
    – Frank
    Aug 28, 2021 at 23:46
  • 1
    $\begingroup$ It might be surprising at first glance, but makes total sense because the weights of positive samples are still $1$. $\endgroup$
    – gunes
    Aug 28, 2021 at 23:48
  • 1
    $\begingroup$ For training by the way, the positives are so rare than I need to balance the training set with a ratio of 1:1 for positives:negatives, or maybe 1:5, or the classifier "does not see anything" and reverts to always predicting a negative - I guess I could also change the loss function. $\endgroup$
    – Frank
    Aug 28, 2021 at 23:59
0
$\begingroup$

I think your recall equation is incorrect and you would need to scale up the FN count. You are stratifying your sample by the classifier label, and need to reweight when computing the metric.

As a toy example, imagine that your classifier had 100% precision and 50% recall, and the true population looked like this:

10k (TP) 0 (FP)
10k (FN) 99.98M (TN)

If you sample all 10k events where the classifier gave a positive label (i.e. sampling the entire first row above), you would have TP = 10k, FP = 0.

If you sample 100k examples where the classifier gave a negative label, on average you would obtain roughly 100k * (10k/99.98M) = 10.002 FN.

Using your equation, your recall would be 10k/(10k + 10.002) = 99.9%, which is clearly not correct.

In the population, the positive:negative label ratio is 1:9,999, and you sampled at a ratio of 1:10. This means that you'll need to upweight the FNs by a factor of 999.9.

Doing this would give you a recall of 10k/(10k + 10.002 * 999.9) = 50%.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.