1
$\begingroup$

I'm trying to design a system that does some binary classification. The cost of a false negative is high, so I want to ensure that I've tested thoroughly enough to minimize this chance. I would like the system to have a false negative rate of less than 1% with 95% certainty. If I expect the true positive rate to be small (around 2%), how should I calculate the required population size to reach this level of confidence? Is the one-sided test procedure the right equation to use here?

$$ N \geq \left ( \frac{z_{1-\alpha} \sqrt{p_0(1-p_0)} + z_{1-\beta} \sqrt{p_1(1-p_1)}}{p_1-p_0} \right ) ^2 $$

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! To avoid confusion, could you please include a link to your source for that sample-size estimate? Note that the false-negative and true-positive rates aren't determined by the model itself, but by the probability cutoff that you choose to distinguish the groups after modeling. You can make a false-negative rate as low as you'd like by adjusting that cutoff, at the cost of adding false positives. With a good probability model, you can choose a probability cutoff to balance the relative costs of all types of (mis-)assignments. $\endgroup$
    – EdM
    Commented Jul 20 at 2:52
  • $\begingroup$ I think you need to reformulate this in terms of MDE (Minimum Detectable Effect). Then go from there. $\endgroup$
    – usεr11852
    Commented Jul 20 at 2:55

1 Answer 1

Reset to default
3
$\begingroup$

The problem is that you can reduce the false-negative rate to 0 if you just assign all cases to the positive class. That's independent of a model and presumably not what you want.

What you need is a reliable, calibrated model of the probability of the (rare) outcome as a function of predictors. Then, based on that model, choose the probability cutoff that best balances the relative costs and benefits of different (mis)assignments of class membership. This answer outlines the considerations, with links to further reading. This answer illustrates how to take benefits (negative "costs") of correct assignments into account.

The formula you show does not readily apply to your situation. It's for designing a study to test whether two groups differ in the probability of some outcome, not for what you want to know: the probability that some model (based a set of predictors) predicts that outcome with desired performance.

There are tools for power analysis to inform design of a study to detect the association of a predictor with outcome in logistic regression; see this page among others. Those tools require knowing something about the association of that predictor with the other predictors in the model.

But even that's not what you seem to want to do. You would like to know the sample size needed to assure a desired level of model performance. That depends on details specific to the subject matter. Increasing the size of a sample might make your estimates of associations of predictors with outcome more precise, but if the predictors aren't adequately associated with outcome (at least in your model) then increasing sample size won't help much in improving probability estimation. If you have a cost-based probability cutoff in mind for the ultimate classification task, you might consider combining multiple models with Targeted Maximum Likelihood Estimation.

In practice, however, it's often the case that a model alone isn't used to make decisions. Think about how the model will be used. This seems like a situation that would benefit from consultation with a local statistical expert with whom you can discuss the details of your subject matter.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.