I created a rule based classification model for flagging emails about a certain topic. The process for creating that model, was to look at contents and flagging the email if certain terms are used.
I now want to estimate the number of flagged emails for 2019. For this estimation we took 400 emails at random during 2019 and had the following results:
- True Positive: 40
- False Positive: 10
- True Negative: 300
- False Negative: 50
The easiest way to estimate the volume of flagged emails for 2019 would be to use the proportion found on the sample:
- (40 + 50) / 400 = 0.225
- Confidence interval with N = 400, p = 0.225 and z = 1.96 --> +- 0.04
Imagine 100,000 emails in 2019, the estimate would be: 22,500 +- 896 flagged
I was thinking I could use the model to be more accurate:
- Coefficient ratio (TP + FN) / (TP + FP) = 1.8
- Bootstrap 1000 times with a resample of 400 emails and 95% confidence : +- 0.06 (fictional number)
Imagine 12,000 of the 100,000 emails tagged by the model in 2019, the estimate would be: 21,600 +- 720
Can I apply method 2 ? Or have I missed an essential assumption?
Which method would be most appropriate for estimating the number of flagged emails in 2019?