RFE on small dataset: What kind of cross validation should I use?

Question

I’m using recursive feature elimination (RFE) to rank features. My dataset contains 50 observations and 20 predictors. Is there a specific type of cross-validation method I should be using to estimate error? So far I’ve tried leave-one-out, repeated k-fold (100 40/10 splits), and bootstrap. Each method gets me different results.

Answer 1

The difficulty with small sample size and data-driven model optimization (such as recursive feature elimination) is that the optimization needs sufficiently certain estimates of the target function (importance).

While (in)stability and thus uncertainty depends on the figure of merit, in general, the fewer cases you have to calculate the figure of merit, the more uncertain it gets. One symptom of this is that the optimization "results" become unstable.

Since you are in a situation where the results are unstable, you have several options:

• Stabilizing the optimization by measuring random uncertainty (that is possible e.g. via bootstrap or repeated CV) and incorporating this into the elimination decision.
• Model aggregation: keeping a number of optimization results and build an aggregated model from this. The aggregated model with likely contain many features, though.
• Eliminate features by external knowledge (i.e. knowledge gained without this data set)
• Use a less uncertain (less noisy) figure of merit
• LASSO regularization also keeps a subset of features only

---

In any case, you crucially need a proper validation (verification) of the performance of your final model. E.g. nested CV, an independent outer out-of-bootstrap performance estimate, independent test set etc.