1
$\begingroup$

I have a binary classifier with 94% accuracy on unknown test data. I use that model to classify samples from a large population in order to infer the proportion of positives within the population. I use Clopper-Pearson to construct a confidence interval with a significance level of $\alpha=0.05$, so that the true proportion of positives is contained in my predicted interval 95% of the time.

Now, given my classifier's accuracy of 94%, 4.2% of false negatives and 1.6% of false positives, should I adjust my confidence interval in some way? Intuitively, it makes sense to widen the interval, because of the added uncertainty of my classifier. How would I go about that, while keeping everything statistically correct?

Or how does the accuracy affect the significance level?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.