1
$\begingroup$

Context

I created a rule based classification model for flagging emails about a certain topic. The process for creating that model, was to look at contents and flagging the email if certain terms are used.

I now want to estimate the number of flagged emails for 2019. For this estimation we took 400 emails at random during 2019 and had the following results:

  • True Positive: 40
  • False Positive: 10
  • True Negative: 300
  • False Negative: 50

Method 1

The easiest way to estimate the volume of flagged emails for 2019 would be to use the proportion found on the sample:

  • (40 + 50) / 400 = 0.225
  • Confidence interval with N = 400, p = 0.225 and z = 1.96 --> +- 0.04

Imagine 100,000 emails in 2019, the estimate would be: 22,500 +- 896 flagged

Method 2

I was thinking I could use the model to be more accurate:

  • Coefficient ratio (TP + FN) / (TP + FP) = 1.8
  • Bootstrap 1000 times with a resample of 400 emails and 95% confidence : +- 0.06 (fictional number)

Imagine 12,000 of the 100,000 emails tagged by the model in 2019, the estimate would be: 21,600 +- 720

Questions

  • Can I apply method 2 ? Or have I missed an essential assumption?

  • Which method would be most appropriate for estimating the number of flagged emails in 2019?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.