Feature extraction with k-fold elastic net - what is the test data?

Question

I am trying to select features and develop a predictive model.

Imagine I run a elastic net regression where lambda > 0. There are ten predictors, and the coefficients for five of those predictors is set to zero. The other five have coefficients that do not equal zero. This was all completed on the same dataset though the the tuning of alpha and lambda parameters was done using a search grid and k-fold cross-validation.

I am assuming that it is appropriate that the feature selection, model training and model validation were combined when I ran the elastic net and estimated the lambda/alpha values using k-fold cross-validation. However, there should be a test phase too where I go about estimating model performance.

Do I need to run this model using the coefficients on a test set? Or is there another option?

(FYI: I am using caret but this is aimed at being a conceptual question rather that is function-specific)

Answer 1

You should first split your data into training and test sets. The exact split will depend on the amount of data you have but most suggest 70:30.

You should use the training set for your model training, tuning and parameter selection. In caret this is usually via a grid search and cross validation in multiple training and validation sets.

Once you have the optimum model from your test split, you then work out the prediction performance on the test set. This minimises overfitting.

See this question- What is the difference between test set and validation set?