Given that I have an algorithm that classifies data points as 'true' or 'false'. and I want to estimate its FPR, FNR. It is not a supervised model where I start with a large training set of labeled data, so I thought of getting a sample data set and labeling it manually. Labeling is very laborious, so I want the sample set to be as small as possible, what is the formula to calculate the sample size (and obviously, what parameters will it require)?

In particular, I don't have a good estimate about the real ratio of 'true' vs. 'false' in the real world.

Are there better measures than FPR / FNR in such a case?

  • 1
    $\begingroup$ Unclear to me - If you dont have labeled data points how do you expect to estimate FPR and FNR?. $\endgroup$ – yoav_aaa Jan 2 '18 at 14:09
  • 1
    $\begingroup$ I don't have a large set of labeled data points, but I can sample and label manually. I'll clarify $\endgroup$ – IttayD Jan 2 '18 at 14:15

As I understand it, your question is consisted of two smaller questions:

  1. What is the needed sample size to evaluate my classification model(algorithm)?
  2. How do I select my evaluation(labeled) data in a manner that reduce selection bias(so my evaluation reflects real world performance)?

The first question is strongly related to the "samples per features" question.
Where the logic behind answering it is a function of your algorithm complexity. The more complex (more features/variables it uses) the more samples you need. There are many discussions around this topic with no clear cut rule of thumb arising.
A quick search led to these related questions on this topic:
How large a training set is needed?
Number of features vs. number of observations

Answering the second question. First, assure the whole of your data represents the real world (if not, then you need to get more data, or find ways extrapolating the results).
Under the assumption that it is representative, select a subset from the data (with size determined in previous step) that have the same characteristics as the full data set.
You can measure this similarity using goodness-of-fit measure - The Kolomogorov-Smirnov test is widely used.
And this article(section 2.3) is a good reading about this "training" subset selection process.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.