I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one you used to tune the features.
What I'm wondering is how bad of a problem this is. I can understand how it would be really bad for feature selection, since this gives you a huge number of parameters to tune. But what if you're using something like LASSO (which has only one parameter, the regularization strength), or a random forest without feature selection (which can have a few parameters but nothing as dramatic as adding/dropping noise features)?
In these scenarios, how badly optimistic could you expect your estimate of training error to be?
I'd appreciate any info on this--case studies, papers, anecdata, etc. Thanks!
EDIT: To clarify, I'm not talking about estimating model performance on training data (i.e., not using cross validation at all). By "hyperparameter tuning outside of cross-validation" I mean using cross-validation only to estimate the performance of each individual model, but not including an outer, second cross-validation loop to correct for overfitting within the hyperparameter tuning procedure (as distinct from overfitting during the training procedure). See e.g. the answer here.