How to design cross-validation and testing scheme when N is small? I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make predictions. I expect co-linearity between some of the features. If I devote 20 samples to my test set and divide responders and non-responders equally between the training and testing sets, I would get 6-7 responders in the test set. I would perform cross validation to select between LASSO regression models and random forest with a few parameter values for each. Finally, I would train the best model (one standard error rule) on the entire training set, and then apply to the test set.

*

*I understand that N is too small. Is it foolhardy to even attempt classification in this scenario?


*Is it permissible or even advisable to over-represent the smaller class (non-responders) in the test set? Perhaps I randomly select 10 responders and 10 non-responders for the test set. I might have more statistical confidence. With only 6 non-responders in the test set, it might be easier for me to get "lucky" or "unlucky". A bad classifier could accidentally perform well or vice versa. If my classifier were to perform well by some metric, how could I be confident that was not due to chance?


*Should I use leave-one-out cross validation in this scenario? With 40 samples in the training set and 10 non-responders, I could do 5-folds cross validation and randomly select 2 non-responders and 8 responders for each CV fold. But then I am only training on 30 training samples for each split vs. 39 if I do leave-one-out CV.
I haven't actually done this experiment yet. I am trying to get a sense for whether this kind of a project is worth attempting with such a small N, and if so, what problems I may need to be wary of.
 A: Cross validation is not the only way to compare different models. Several model selection criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) can precisely be used to compare different hypotheses when the number of observations is small and all samples need to be used for training. The main concept behind these selection criteria is to penalize the likelihood of a model with its complexity (i.e. with its number of free parameters).
A: 
I understand that N is too small. Is it foolhardy to even attempt classification in this scenario?

Probably, at least as you propose to proceed.
First, for train/test splits to work well, you need on the order of 20,000 cases. You are very far from that. Your best option is to repeat your entire modeling process on multiple bootstrap samples, treat each bootstrap sample as a training set, and evaluate each model against the entire data set as a test set. Incorporating the "optimism bootstrap" will allow you to evaluate the extent of overfitting. That assesses the quality of your modeling process, which is all you can hope to do with a small sample.
For a binary outcome and an unpenalized model, to avoid overfitting you typically need about 15 members of the minority class per predictor. See Chapter 4 of Frank Harrell's course notes. With 20 in your minority class, that's only 1 or maybe 2 unpenalized predictors.
Penalization via LASSO will tend to retain a similar number of predictors, 1 or 2 in this data set. The particular predictors retained, however, are likely to differ from bootstrap sample to bootstrap sample, particularly with collinear predictors. See Statistical Learning with Sparsity, Section 6.2. There's no assurance that those retained in the model on the full data set will be the "best choices" overall, only that they happened to work on this small data set. Is that OK for your application?
Critically, don't use accuracy as your performance criterion. You should work with probability estimates for as long as possible, using log loss or other strictly proper scoring rules. If you do ultimately have to classify, the choice of probability cutoff should take into account the relative costs of false-positive and false-negative classifications. The (often hidden) default of a cutoff at probability of 0.5 for "accuracy" is only appropriate if those are equally costly, which isn't necessarily true.
As you are still in the study design phase, simulate large numbers of cases based on reasonable assumptions about outcomes, their associations with your 20 features, and the correlations among feature values. See how well you can model your known associations with outcome on samples of size 60. That might be the best way to see whether this modeling attempt really is foolhardy.
You might be better off using you knowledge of the subject matter to do data reduction on the set of features, combining them intelligently into a smaller number of combined predictors, rather than relying on the types of automated procedures you envision. Again, see Harrell's course notes for guidance.
A: In addition to the excellent points in Camille's and EdM's answers:
Purpose of the model/study
Whether something sensible can be done or not depends crucially on the purpose of the modeling.
For any kind of real-world application, the random uncertainty on your internal validation results (even with the best resampling) will likely not be useful.
If on the other hand, this is a preliminary study with the purpose of generating a first rough idea to substantiate a grant proposal, that would be perfectly fine.
Random uncertainty in internal validation results
Do a back-of-the-envelope assessment of the situation.
The best resampling cannot get around the fact that you have only 60 cases (forget about hold-out), and only 20 in the smaller class.
Proportions like acurracy, sensitivity, specificity etc. have many disadvantages over the already mentioned proper scoring rules. Right here, however, they have the one advantage that they allow back-of-the-envelope calculations before even preliminary data is available.
(Proper scoring rules can have lower variance than these proportions, but don't expect miracles. Plus, overfitting leading to overconfidence in the predictions, thus predicted probabilities too close to 0 or 1, increases variance uncertainty)
With 20 patients in the negative class, specificity (the probability of correctly recognizing negative class/non-responders) will have 95 % confidence intervals (calculated via Bayes method with uniform prior):

If this uncertainty is (unjustifiably) too large for your purpose, then don't even start experiments now. Instead, you can use such confidence interval estimations to calculate a more sensible number of cases and recruit them.

*

*Note that this includes only random uncertainty that stems from the limited number of cases. Resampling in small-sample-size situations or across optimization processes there will be additional variance due to model instability.


*As I said, (strictly) proper scoring rules feature lower variance among their advantages over proportions. However, you may find that you are forced to communicate final internal validation results in the form of proportions, and then you're back to these large confidence intervals.


*Looking at the variance of a figure of merit between cross validation folds gives you a funny mix of $\frac{n}{k}$ case-to-case variance and model instability variance.
(avoiding) Data-driven model optimization
These confidence intervals should also give you an idea of how large the observed difference in performance between candidate models should be to allow a data-driven choice. (Remember, choosing between multiple models -> multiple comparisons). The likely conclusion is that you cannot do data-driven model optimization with so few cases. Not even if you switch to a nested resampling strategy, and while AIC, BIC etc. spare you the inner resampling, they also cannot do miracles and will basically tell you that you cannot afford a complex model based on these case numbers.  However, they may be more convenient for formulating a decision strategy for the model complexity that is then evaluated in the resampling.
Other than that, your best bet is to include as much external knowledge as possible and reduce the dimensionality of your data before even looking at it. I.e., avoid the data-driven optimization and replace it by decisions you take.
E.g. if you know you have 3 clinical parameters that are highly correlated, either decide for now to go with one (i.e. decide which one) or if you prefer with their average or sum (or difference for negative correlation) and then swallow the bill for this professional decision.  The cost (in terms of performance) of being undecided will likely be heavier, because data-driven optimization costs sample size...
Or decide to go for the LASSO model using 5 parameters, without looking whether 4 or 6 look better.
The random forest you mention takes yet another strategy, and may be your best bet if you do not have sufficient external knowledge. The rF tries to "average out" model instability. The beauty here is that default parameters often yield a sensible model, so don't even think of optimizing them. If you want to be on the safe side, use more trees though: there you trade of computational resources for a possible gain in predictive performance, but you do not risk worse predictive performance by using more trees.

Leave-one-out
... is almost never advisable since it conflates surrogate model and test case, depriving you of the possibility to check stability of the predictions (there are more disadvantages to LOO, but that alone is sufficient for me to avoid it).
Instead, go for a resampling scheme like bootstrap or repeated k-fold cross validation (say, 50x10-fold or 25x20-fold) that allows you to measure instability, possibly stratified.
Is it permissible or even advisable to over-represent the smaller class (non-responders) in the test set?
Adjusting relative class frequencies is IMHO advisable only if you correct your data to better represent the application-scenario specific relative class frequencies.
All other adjustments, in particular towards better balance in the training data may lead to nice-looking results at the first glance, but that will usually come at the cost of worse performance under application working conditions.
Instead of adjusting relative class frequencies in the test data, it is IMHO better to adjust the resulting figures of merit to application-relevant relative frequencies.

Finally, be happy: I often have to work with smaller sample size and at the same time much higher dimensionality (p in the 100s or 1000s). (But I can typically use repeated measurements and external knowledge about the physics and chemistry behind the data - which makes modelling possible - even if it does not help with the uncertainty of the internal validation)
