Let's assume I want to construct a regression model to predict a specific outcome variable but I don't have enough data to do a proper train-test set split (n = 200). I have 7 predictor variables (some of which are correlated) so I decide to use and elastic net (Glmnet/caret).
I first run a model without cross validation to determine the optimal alpha and lambda values for the model (i.e parameters that return lowest RMSE value).
model_tune <- train(outcome_variable ~ ., data = data, method = "glmnet", tuneLength = 10, preProcess = c("scale", "center"))
I then extract the alpha and lambda values for the best fit model using:
I then rerun the model using the best fit model's alpha and lambda values, but include cross validation to "estimate" or "simulate" how the chosen model would have performed on "unseen" data.
model_estimate <- train(outcome_variable ~ ., data = data, method = "glmnet", trControl = trainControl("cv", number = 10), tuneGrid = expand.grid(alpha = 0.5, lambda = 2.3), preProcess=c("scale", "center")) model_estimate$results
I think I'm having a hard time figuring out where cross validation comes in regarding the evaluation of the model accuracy vs parameter tuning and what exactly I should be reporting.
- Using cross validation am I just going to get the average RMSE value of the resampled models and that's what I should report?
- What about the coefficients for my cross validated model? Where do they come in? Can I just report the coefficient found in the best fit model? Should they also be based on the averages for the resmaples?