I have only a small dataset. I want to 1. select the most predictive features out of a large candidate pool and 2. get an estimate of their expected predictive performance.
In the elements of statistical learning (page 245ff), the authors stress the importance of including variable selection within the cross-validation loop for obtaining unbiased estimates of expected out-of-sample performance.
However, the estimates of model performance obtained in this manner will not be for one defined set of features, as each cross-validation fold or repetition may lead to the selection of different features. I am, however, not interested in model performance averaged over different sets of features. I want to obtain one set of "best" features and the performance I can expect conditional on them in independent datasets.
Do I have any options in simultaneously getting both 1. and 2.?