I am working on a problem where it is expensive to label data and I have sampled a small subset of the available data and labeled it.
My classifier is a binary classifier that I use with the hope of removing samples that are "false" but keep samples that are "true".
My question is: how well does the classifier performance generalize to the full population?
I sample 500 data records and label them (uniform sample).
True | False 180 | 320
If I assume a binomial distribution I can calculate (e.g. using R) a confidence interval as such:
> binom.confint(180,500, conf.level = 0.95, methods="exact") method x n mean lower upper 1 exact 180 500 0.36 0.317863 0.4038034
If I then use a classifier and the performance is the following:
=== Confusion Matrix === a b <-- classified as 150 30 | a = related 33 287 | b = unrelated
I know how I can calculate precision and F measure from this result, but what is the error?
How well does this generalize to the whole population? Can we make any error estimation with the help of the above confidence interval of the binomial distribution?
I have been doing some calculations where I assume that the true-positive and false-positive rates from the classification hold in general. But I know that this is more or less a back-of the envelope type estimation.