I am trying to select features and develop a predictive model.
Imagine I run a elastic net regression where lambda > 0. There are ten predictors, and the coefficients for five of those predictors is set to zero. The other five have coefficients that do not equal zero. This was all completed on the same dataset though the the tuning of alpha and lambda parameters was done using a search grid and k-fold cross-validation.
I am assuming that it is appropriate that the feature selection, model training and model validation were combined when I ran the elastic net and estimated the lambda/alpha values using k-fold cross-validation. However, there should be a test phase too where I go about estimating model performance.
Do I need to run this model using the coefficients on a test set? Or is there another option?
(FYI: I am using caret but this is aimed at being a conceptual question rather that is function-specific)