I am going to make a distinction between predictive modeling (where prediction accuracy is important) vs. inference modeling (where understanding relationships and meaning of predictors is important).
For predictive modeling, it is good practice to have a held out test set, and to tune your hyper-parameters on the cross-validated training set for each algorithm (random forest vs. svm vs. linear regression etc.). After tuning your hyper-parameters you can then train each algorithm on the full training set and then compare each algorithm's accuracy on the held-out test set to see which algorithm may be the most accurate for your problem.
However, if your goal is not prediction accuracy, but inference (by using linear regression), is there a way to prevent overfitting? I understand that I can use cross-validated variable selection, regularization (lasso, ridge), and check for multicollinearity, but do I do that all the whole data set? Or is a held-out test set still appropriate? (I don't really understand the purpose of a held-out test set if I am just using one algorithm - linear regression).
One idea that I had was to make multiple B bootstrapped datasets, and for each B datasets, go through variable selection and regularization techniques, and perhaps average the mean and standard errors of each B datasets to see what variables are significant. Is this method valid? Do I still need to cross-validate within each B datasets during my variable selection technique?
