I have a machine learning classification problem where 0.05% of the population (N = 100k) is of the positive class. It is important that I don't misclassify these positives (aka I want to minimize the number of false negatives). I want to say with 95% confidence that my machine-learning model does not misclassify any positives. How large should my sample size be?
2
-
2$\begingroup$ Welcome to Cross Validated! If false negatives are completely unacceptable, then you can’t ever classify as a negative, so your model is extremely simple: when it is presented with data, it classifies as a positive case. How would that work for your application? $\endgroup$– DaveMay 17 at 11:24
-
$\begingroup$ Am I understanding correctly? There is a population of 100k points, you know that 50 of them are true positives, but you don't have access to the full population data. You want to subsample the population and use the subsample to train a classifier that will be correct on all 50. $\endgroup$– usulMay 17 at 21:55
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Classify everything as positive. You now are 100% sure you do not misclassify any positives. Problem solved.
I would thus recommend you be a little more detailed on the costs of misclassification, where you will likely need to keep subsequent decisions made on the basis of the classification in mind. The beginning of this answer may be helpful.
"Does not misclassify any positives" is an extremely hard standard to reach. Real life always contains edge cases and Black Swans. "Zero defects" sounds good on paper, but in practice it will mean that your sample will need to encompass the entire population, because you can never be sure that one of the data points you did not sample will throw off your model.
Once you have a good notion of the costs of misclassifications, you can start simulating with different sample sizes and recording the resulting costs your model causes. Beyond some sample size, these costs will hopefully stay more or less flat. Weigh this sample size against the costs of collecting and processing data.
There is likely no closed formula that tells you the required sample size, because it depends hugely on your data - if you have a predictor that reliably tells you which class an instance belongs to, you need no sample size at all. Conversely, your problem may be so noisy that the cost you want to reach is simply not achievable: How to know that your machine learning problem is hopeless?
answered May 17 at 11:29
-
$\begingroup$ I mostly like this answer. However, your first and third points seem to contradict each other. No false negatives is extremely easy to achieve, granted, at what might not be an acceptable cost. $\endgroup$– DaveMay 17 at 11:33
-
1$\begingroup$ @Dave: I implicitly assumed that point 1 was unacceptable. In my experience, few people will be happy when I propose a "solution" like this. $\endgroup$ May 17 at 11:34
$\begingroup$
$\endgroup$
This is like a test for the probability parameter of a Bernoulli distribution.
It is a bit tricky when you test the null hypothesis that the parameter is equal to zero. In that case, given the hypothesis is true, then the only outcome with non-zero probability is zero cases. The p-value is either 0 or 1, independent of the sample size. So you are testing effectively a degenerate distribution.
You can however compute a 95%-confidence interval like the rule of three and demand that the upper boundary, $3/n$, should be below some level.
answered May 17 at 11:34