I am trying to generally determine sample size needed for logistic regression when using cross-validation.
The "1 in 10" rule for sample size in logistic regression is that you need about 10 cases per covariate in your minority class. So, for example, if you had a sample of 2500 people, and 500 people died, you could fit 50 variables reliably to this model since your minority class has 500 observations, and you need 10 observations in the minority class per variable. Generally, this rule of thumb can be given by:
$N = $ $\frac{10\times{k}}{p}$ ,
where $k =$ The number of covariates in the model
and $p =$ The porportion of your minority split.
So, in the above example with $ k = 3$ and $p=0.2$,
$2500 = $ $\frac{10\times{50}}{0.2}$ ,
Cross validation (to prevent overfitting) would divide my sample of 2500 into a random permutation of the sample set. In this case, using 5 fold-cross validation and using a 20%/80% train/test split, would leave me with training sets of 2000 on each fold (and even then, I'd have to hope/assume that the class splits were preserved). If the class splits are preserved, then I'd have a training sample of 2000, with 1600 in my largest class and 400 in my minority class. Then, I could only fit 40 variables reliably to this model using the rule of thumb.
So, should I be adjusting for cross validation requirements in my sample size estimates? And what if the cross validation did not preserve the class splits? Would some of the models fit in the folds potentially not be reliable? Or generally, does cross-validation lessen the need to apply the 1 in 10 rule since it is supposed to prevent over fitting in the first place and provide more reliable outputs?
Here's a visual reference for how 5-fold cross validations divides the data up: