I have a binary classifier with 94% accuracy on unknown test data. I use that model to classify samples from a large population in order to infer the proportion of positives within the population. I use Clopper-Pearson to construct a confidence interval with a significance level of $\alpha=0.05$, so that the true proportion of positives is contained in my predicted interval 95% of the time.

Now, given my classifier's accuracy of 94%, 4.2% of false negatives and 1.6% of false positives, should I adjust my confidence interval in some way? Intuitively, it makes sense to widen the interval, because of the added uncertainty of my classifier. How would I go about that, while keeping everything statistically correct?

Or how does the accuracy affect the significance level?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.