Selecting features and estimating their out-of-sample performance with cross-validation

Question

I have only a small dataset. I want to 1. select the most predictive features out of a large candidate pool and 2. get an estimate of their expected predictive performance.

In the elements of statistical learning (page 245ff), the authors stress the importance of including variable selection within the cross-validation loop for obtaining unbiased estimates of expected out-of-sample performance.

However, the estimates of model performance obtained in this manner will not be for one defined set of features, as each cross-validation fold or repetition may lead to the selection of different features. I am, however, not interested in model performance averaged over different sets of features. I want to obtain one set of "best" features and the performance I can expect conditional on them in independent datasets.

Do I have any options in simultaneously getting both 1. and 2.?

Related question

Answer 1

I'd like to add one practical recommendation here:

If you want to use e.g. cross-validation estimates of model performance, I recommend to build and cross validate a model using pre-specified settings according to your knowledge about the data type, application problem and the type of model. For this "starting" model, estimate performance and confidence intervals of the performance (for proportions like classification accuracy etc. use binomial confidence intervals).

The next step then would then to step back and judge whether you expect to achieve an increase in performance that can be shown to be an actual increase also when taking into account the variance of the performance estimates.

Example:

Say, you do binary classification of 10 patients (doesn't matter how many repeated measurements per patient you have as the patients are the statistically independent cases and thus your sample size). Assume you oberve 8 correct for you not-optimiized model. A 95% confidence interval for the observe 80% correct of 10 patients ranges approximately from 52 to 96 %. In other words, basically from guessing to perfect (or at least: very good).

You could have known that beforehand: a perfect model would yield 10 correct out of the 10 patients. Yet the 95% confidence interval would have a lower limit of ca. 70%. The upper limit of the 95% c.i. for a "guessing" model with 5 correct out of 10 patients is ca. 75 %. Actually, even with a paired test you cannot distinguish the apparently perfect from the apparently guessing results.

Answer 2

As above. But to add some emphasis:
(1) The purpose of internal validation as described it to validate the modeling process, and not the model. That is a fairly important objective. But they are related. Validating the modeling process given you an estimate of how overfit your model may be.
(2) The internal validation should thus provide you with a measure of your "optimism". This can be used to adjust your apparent test metric.

Example for bootstrap validation:
Apparent area under the ROC curve of the model: 0.77
Mean area of 500 bootstrap samples:0.772
Mean area of 500 tests in original: 0.762
Optimism in apparent performance: 0.01
Optimism-corrected area: 0.76

(3) Or you can use your test dataset, with the caveat that this dataset should be fairly large if it is to provide stable estimates.
(4) Selecting the "best features" is a non-trivial task. I would be cautions about any statements regarding this, especially in smaller datasets.
I'm sure there is a section in textbook above on interpreting results in high dimensions. This could easily be applied to small datasets.
One of the benefits of bootstrapping or cross-validation is that is has the potential of providing you with some indication of the variability in variable selection.