R- Is there a sample size that is “too small” for nested k-fold cross-validation? I'm sort of new to model building using cross-validation, but am currently working with a relatively small data set (n = 70) where I would like to try and apply this technique.
Based on my understanding, one approach to doing this would be to perform a nested k-fold cross validation where the inside & outside folds aim to specify the model and model hyperparameters, respectively. The number of folds seems to be empirically determined to be k = 5 or k = 10 in order to optimize any bias-variance relationships.
My question(s) is/are this: Is there a sample size that is effectively "too small" to work with this approach? With 70 subjects, two cross-validation steps with 10 folds each would effectively lead to small partitions with 0-1 subjects. Also, would a different cross-validation technique work "better" in this situation (e.g. LOOCV)? And if so, does anyone have any example code or resources that I may refer to in order to get started?
I feel that the answer ultimately "depends" on both the data itself as well as how stable the derived models may be between folds.
 A: Summary: Yes there is a sample size that is too small for nested $k$-fold cross validation - but probably not in the way you think: the bottleneck is probably going to be in the verification/testing. The uncertainty on your test results depends fundamentally on the absolute number of tested cases (in the absence of any gross errors such as the CV splitting not being independent). This limitation is very fundamental and cannot be overcome by using another cross validation scheme (or, much worse, single splits).

In your scenario with binary 0/1 predictions (as opposed to continuous scores), your loss function will likely be a proportion of tested cases and these have high variance (they follow a binomial distribution).
Assume as a thought experiment that the outer CV result is 10 % error (63 of 70 correct). This observed 90 % accuracy based on 70 tested cases in total has a 95 % confidence interval ranging roughly from 80 - 95 %. If this uncertainty in the final model performance is too large to be acceptable for your application, there is no way to solve your task with just 70 cases. 
If you nest 10-fold outer / 9-fold inner cross validation, the hyperparameter tuning in the inner loop has 63 cases.
If you calculate e.g. accuracy on the basis of 63 CV-tested cases, a single paired comparison (McNemar's test, you can play around with R's mcnemar.test() to get a feeling for the situation) will need to have roughly a difference of 0.1 (10 %points) in accuracy to be significant. If you do multiple comparisons, the difference will need to increase. And that doesn't even account for uncertainty due to too complex models being unstable.
But a difference of 0.1 in accuracy is quite large for many applications, and very coarse for most hyperparameter optimizations. 
You can find more information on these questions in our papers:


*

*for the confidence intervals: Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

*For measuring instability: Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations Anal Bioanal Chem, 2008, 390, 1261-1271.
DOI: 10.1007/s00216-007-1818-6

Recommendation:


*

*Fix your hyperparameters by external knowledge if at all possible (e.g. deciding that with 63 cases in the training, you cannot afford more than 5 or 6 features and accordingly fix the Lasso so that 5 or 6 features are used). Do repeated cross valiation to also check model stability for this (as you don't tune the Lasso parameter based on predictive performance, you don't need the inner CV).

*If that is not possible, and you have to go for nested cross validation, repeat both inner and outer cross validations and estimate model stability from that. Unless the apparent optimum is really unstable, go for the least complex model that is not significantly worse than the apparently best one (do a bit of correction for multiple comparisons). If the apparent optimum has non-negligible instability, add some more "safety margin" for that. (This is my personal variant of what is otherwise known as the one-standard-deviation-rule)
