What to report in cross validated elastic net regression?

Question

Let's assume I want to construct a regression model to predict a specific outcome variable but I don't have enough data to do a proper train-test set split (n = 200). I have 7 predictor variables (some of which are correlated) so I decide to use and elastic net (Glmnet/caret).

I first run a model without cross validation to determine the optimal alpha and lambda values for the model (i.e parameters that return lowest RMSE value).

model_tune <- train(outcome_variable ~ ., data = data, method = "glmnet",
                    tuneLength = 10, preProcess = c("scale", "center"))

I then extract the alpha and lambda values for the best fit model using:

model_tune$bestTune

I then rerun the model using the best fit model's alpha and lambda values, but include cross validation to "estimate" or "simulate" how the chosen model would have performed on "unseen" data.

model_estimate <- train(outcome_variable ~ ., data = data, method = "glmnet", trControl = 
                    trainControl("cv", number = 10), tuneGrid = expand.grid(alpha = 0.5, lambda = 
                    2.3), preProcess=c("scale", "center"))

model_estimate$results

I think I'm having a hard time figuring out where cross validation comes in regarding the evaluation of the model accuracy vs parameter tuning and what exactly I should be reporting.

1. Using cross validation am I just going to get the average RMSE value of the resampled models and that's what I should report?
2. What about the coefficients for my cross validated model? Where do they come in? Can I just report the coefficient found in the best fit model? Should they also be based on the averages for the resmaples?

Answer 1

You might be best off using the bootstrap to validate your modeling process.

In outline, you repeat the entire modeling process--including the determination of optimal $\alpha$ and $\lambda$ values by cross validation--on multiple bootstrap samples of your data, then evaluate each model's performance on the entire data set. That mimics the process of sampling from the original population, modeling each sample, and then testing on the population. See section 5.3.5 of Frank Harrell's class notes.

Then you report the original model based on the full data sample along with the performance of the bootstrapped modeling process as an estimate of how well you have captured the modeled relationships in the population.