note: I read a lot of the questions already posted on this topic, but still have some confusion.
I want to perform feature selection and model selection for multiple models e.g. Random forest (RF), Support vector machine (SVM), lasso regression. There seem to be a few ways to do feature selection (fs) or hyper parameter optimization (hpo) through cross validation (cv). My data set is n~700 (sample size) and p = 272 (number of features). However, adding another set of features could increase p to ~20272.
My current plan is the following:
- Run whatever resampling method (k fold or Monte carlo) to get different splits of pseudo test and training data.
- In each iteration of resampling:
- Run feature selection on pseudo training data
- Increment counts for which top variables are selected
- Train model using those features on pseudo training data
- Get estimate for how well it does by testing on pseudo test data
- Now we can select our feature set by taking the top k selected variables after however many iterations of resampling.
- Using our selected feature set, run hpo for all of our models of interest in same manner as above (getting estimation of error for models with different hyperparameters trained on pseudo train data and tested on the pseudo test set, then taking the hyperparameter that leads to lowest [pseudo] test error)
- Now we have our selected feature set and optimal hyperparameters. Last step is to build models with whatever optimal hyperparameters and selected feature set on full training data, and get error on the test set.
I have several questions:
Is it ok to do fs and hpo separately through many resampling phases? As well, is it recommended to run fs again on overall training set after running these rounds of cv (then, the cv purpose would have been to verify that we select the same features most of the time)? Likewise, should we do hpo again on overall training data too (to verify that overall, the same hyperparams are selected), and if so, should i use something like k fold or monte carlo to get those final validation errors?
My other question is should i merge fs and hpo in the same resampling phase? If we had an initial guess for possible subsets, we can treat it like another hyper parameter, but for my case we can’t try all 2^|p| subsets so we need some kind of initial filtering. So, is it ok to do initial filtering first? If so, should i do it on an iteration of the k fold or monte carlo sampled training data? Wouldn’t this selection be bias with respect to later resampled evaluations?
Please let me know if I am being unclear or if I am doing something wrong/not recommended.