Is it reasonable to run CV on whole data (in a regression model) when the sample size is small My sample size is 550, and I build a regression model with 80 features. In particular, I am using LASSO and ridge regression with 5-fold leave one-out cross-validation. To evaluate the models, I am using correlation (between the predicted and actual value) and mean absolute error (MAE). The correlation I obtain is around .5, and the MAE is around .8. The value of the variable that the model predicts is in the range of 0-8.
My results seem to be promising (but not perfect). Since the sample size is small and the feature set is large, I wonder if it is reasonable to run CV on whole data and evaluate the performance of the model based on this? I am working on an academic publication, and I wonder if this is an acceptable method (with limitations reported). Or, should I use bootstrapping to resample the data, and apply CV on only the training set? I appreciate any insights.
 A: It seems that you have performed ridge regression and LASSO, choosing the penalty parameter values by 5 repetitions of leave-one-out CV, with either the correlation between observed and predicted or the mean absolute error (MAE) as your evaluation measure. Your data set actually isn't that small relative to the number of features as you are using penalized regressions, but you can do a bit better in your model building and evaluation.
First, you should consider whether either the correlation coefficient or MAE are the best measures for choosing model parameters. I think that many would prefer to use the mean-squared error, as that is what both ridge and LASSO start with as their errors before adding the penalization term on the coefficients.
Second, once you have chosen your measure for fit, to choose your penalty parameter values you might be better off doing true 10-fold cross validation to find the best values. Leave-one-out CV can be somewhat noisy, and 5 repetitions might not be enough to overcome that.
Third, what you can perhaps best accomplish is to validate the model building process by repeating your entire process (including your cross-validation to choose your penalty parameter values) over multiple bootstrap samples of the original data, and examining your evaluation measure, over those multiple models, on your original data set. That gives you (and your audience) a good estimate of how well your process would have behaved if you had the chance to take multiple samples from the population. Note that for LASSO you will almost certainly select different features in each of the bootstraps; you might find it informative to look at that issue directly.
This thread and this thread may also be useful. ISLR (Chapters 5 and 6), ESLII (esp. Chapter 7) and Harrell's rms course notes provide more details and examples.
