1
$\begingroup$

I am going to make a distinction between predictive modeling (where prediction accuracy is important) vs. inference modeling (where understanding relationships and meaning of predictors is important).

For predictive modeling, it is good practice to have a held out test set, and to tune your hyper-parameters on the cross-validated training set for each algorithm (random forest vs. svm vs. linear regression etc.). After tuning your hyper-parameters you can then train each algorithm on the full training set and then compare each algorithm's accuracy on the held-out test set to see which algorithm may be the most accurate for your problem.

However, if your goal is not prediction accuracy, but inference (by using linear regression), is there a way to prevent overfitting? I understand that I can use cross-validated variable selection, regularization (lasso, ridge), and check for multicollinearity, but do I do that all the whole data set? Or is a held-out test set still appropriate? (I don't really understand the purpose of a held-out test set if I am just using one algorithm - linear regression).

One idea that I had was to make multiple B bootstrapped datasets, and for each B datasets, go through variable selection and regularization techniques, and perhaps average the mean and standard errors of each B datasets to see what variables are significant. Is this method valid? Do I still need to cross-validate within each B datasets during my variable selection technique?

$\endgroup$
3
$\begingroup$

The simply answer is yes. If you use variable selection to determine which IVs to include in your model (so as to control for these IVs) you will introduce selection effect (bias) and therefore cannot in any straight-forward way trust the results of inference post variable selection. See Leeds "Model Selection and Inference: Facts and Fiction."

Data splitting is the simplest way to avoid this, so unless you're working with a small data set, just split it. There are much more complicated ways to try to avoid selection bias, but someone else will have to explain them.

For starters see this slide deck: http://joshualoftus.com/turing/shorttalk.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.