Logistic Regression 1 in 10 for Sample Size and Cross Validation

Question

I am trying to generally determine sample size needed for logistic regression when using cross-validation.

The "1 in 10" rule for sample size in logistic regression is that you need about 10 cases per covariate in your minority class. So, for example, if you had a sample of 2500 people, and 500 people died, you could fit 50 variables reliably to this model since your minority class has 500 observations, and you need 10 observations in the minority class per variable. Generally, this rule of thumb can be given by:

$N = $ $\frac{10\times{k}}{p}$ ,

where $k =$ The number of covariates in the model

and $p =$ The porportion of your minority split.

So, in the above example with $ k = 3$ and $p=0.2$,

$2500 = $ $\frac{10\times{50}}{0.2}$ ,

Cross validation (to prevent overfitting) would divide my sample of 2500 into a random permutation of the sample set. In this case, using 5 fold-cross validation and using a 20%/80% train/test split, would leave me with training sets of 2000 on each fold (and even then, I'd have to hope/assume that the class splits were preserved). If the class splits are preserved, then I'd have a training sample of 2000, with 1600 in my largest class and 400 in my minority class. Then, I could only fit 40 variables reliably to this model using the rule of thumb.

So, should I be adjusting for cross validation requirements in my sample size estimates? And what if the cross validation did not preserve the class splits? Would some of the models fit in the folds potentially not be reliable? Or generally, does cross-validation lessen the need to apply the 1 in 10 rule since it is supposed to prevent over fitting in the first place and provide more reliable outputs?

Here's a visual reference for how 5-fold cross validations divides the data up:

Answer 1

• First of all: is there really any question whether you should use 40 or 50 variates?
  What do you expect of the 10 features you consider adding?

---

• The 10 cases per variate rule is a very rough rule of thumb that intends to safeguard you against instability.

• However, you can always measure whether your training is stable, e.g. with iterated/repeated cross validation.See e.g. here for some references

• Using cross validation for estimating generalization performance, you implicitly make the assumption that nothing changes substantially when leaving out a few of your training cases. So nothing should be changed between training of your final model and training of the surrogate models.

• If you think that this assumption may be violated (because leaving out 20 % is rather a lot), you can go for higher k, say, 10-fold or 20-fold cross validation.

• Of course "same training procedure" doesn't answer your question as you can still formulate a training procedure that determines number of features as function of available cases.

• Again, the solution is to use your knowledge about the application/task at hand to decide.

---

does cross-validation lessen the need to apply the 1 in 10 rule since it is supposed to prevent over fitting in the first place and provide more reliable outputs?

This is a misconception: cross validation does not prevent overfitting. It can only tell you that you did overfit.

---

what if the cross validation did not preserve the class splits? Would some of the models fit in the folds potentially not be reliable?

If so, cross validation will tell you that you cannot expect great performance. And if that's the result on 80/90/95 % of the training data, would you really rely on the 20/10/5 % additional samples to change "rescuing" your model? I wouldn't.