K-fold validation for logistic regression in R with small sample size I used SPSS to develop a logistic regression model from 274 cases. The final model uses 6 independent variables to predict a binary dependent variable with an event rate of 18%. The model is purely descriptive and has not been validated internally or externally.
I am learning R and want to re-develop the model in R. I am interested in validating the model using either K-fold or leave-n-out cross-validation. What are the strengths and weaknesses of these approaches for my model / dataset? Is it valid to use these methods with a small sample size like mine?
 A: Given a dataset with $N$ records, leave-n out cross validation and K-fold cross validation are equivalent for $n=\frac{N}{K}$.
Technically, the least elements you leave out the better validation will be, in a sense that the models that you build will be the closest to the model you would compute with the full dataset, and therefore your estimate of the performance will be better as well. At the same time, when you leave out only few elements you have to recompute your model more times: for leave-one-out cross validation you retrain your model $N$ times, which can take a huge amount of time. 
All and all, the choice of the number of folds will depend on the size of the dataset and on the learning curve of your algorithm given the size of the dataset. In general, for a dataset as small as yours I would use a high $K$, or even $K=N$ and do leave-one-out CV, as the training of the model takes a short time, while doing few folds might put some bias in your estimate.
A: The sample size is barely sufficient for estimating a model with no covariates, i.e., for estimating the intercept of the logistic model.   But cross-validation, like bootstrapping, can still be helpful here.   Be sure that you do 100 repeats of the cross-validation procedure so that the result will not be dependent on the luck of the split.  And be sure to validate a proper accuracy scoring rule (e.g., Brier score or pseudo $R^2$).  But do note that resampling procedures require that all supervised analysis steps are repeated afresh for each model fit.  So the procedure assumes your model was completely pre-specified without looking at Y, or that you program the data analysis (e.g., variable selection) to be repeated afresh for each resample.
