I am working on a problem where it is expensive to label data and I have sampled a small subset of the available data and labeled it.

My classifier is a binary classifier that I use with the hope of removing samples that are "false" but keep samples that are "true".

My question is: how well does the classifier performance generalize to the full population?

Some numbers:

I sample 500 data records and label them (uniform sample).

True | False 180 | 320

If I assume a binomial distribution I can calculate (e.g. using R) a confidence interval as such:

> binom.confint(180,500, conf.level = 0.95, methods="exact")
  method   x   n mean    lower     upper
1  exact 180 500 0.36 0.317863 0.4038034

If I then use a classifier and the performance is the following:

=== Confusion Matrix ===
 a   b   <-- classified as
150  30 |   a = related 
 33 287 |   b = unrelated

I know how I can calculate precision and F measure from this result, but what is the error?

How well does this generalize to the whole population? Can we make any error estimation with the help of the above confidence interval of the binomial distribution?

I have been doing some calculations where I assume that the true-positive and false-positive rates from the classification hold in general. But I know that this is more or less a back-of the envelope type estimation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.