Is there a test that is used to prevent overfitting regression models with large numbers of potential IVs?

Question

I've set up a measurement system that allows me to measure thousands of different elements, all of which have potential to be used in a regression model. My aim is to create a model that can predict a target (dependent) variable. Making the thousands of measurements is cheap. What is expensive is measuring the target variable, and our budget is blown measuring over 80 specimens.

When it comes to variable selection for the predictive model, I have used both a forward and step-wise selection methods, but then found using a best-subsets method (regsubsets in R) produced even better models (higher adjusted R2 and better Mallow's Cp). However, all of these suggest models that are between 13 and 17 variables, and all the tests suggest that each variable is significant (usually p < .001) and the adjusted R2 is up around 0.95. I am quite dubious about this many, with the potential for overfitting. Given the high number of potential independent variables, I suspect that some of the later ones in the model are no better than chance.

To overcome this potential, I previously doubled the number of IVs, populating the second half with randomly generated numbers (entitled "random IVs" for the remainder of the question). I ensured that each subset had at least 25% from the random IVs group, and worked through all IVs selecting until one random IV appeared in the model. My reasoning behind this ad-hock method is that any IV in a truly predictive model should do better than the best random variable, and therefore despite the low p-value, is not significant. The resulting models used 6-7 IVs, with adjusted R2 values around .85 to 0.9.

My question is therefore two-fold. Firstly, is there a better, formalised test or method to prevent this type of overfitting (specifically associated with large numbers of potential IVs to be tested)? Secondly, if this ad-hock method is appropriate, is there a real limit on the number of random variables that is appropriate? (On this second, I can see that using thousands of random variables may effectively be a very high hurdle to jump.)

Answer 1

To Question 1:

Yes, there is! It is called cross-validation (leave-one-out should work for you). The main problem with your current approach is that you select your predictors solely based on how well they predict the outcome within your current sample. This will likely lead to over-fitting and therefore bad performance in new samples. In cross-validation you split your sample into a "training" sample (where you select your predictors) and a "cross-validation" samples (where you test performance) multiple times to circumventing the problem.

To Question 2:

Again, yes, there is! Cross-validation is often used in combination with regularised regression, which help you with selecting/weighing the predictors. The functions you are looking for are: Ridge regression, LASSO, or Elastic net

As far as I can see you are currently trying to solve a prediction problem with inferential methods. The methods and workflow in predictive modelling (aka machine learning) are somewhat different from classic inferential statistics. You may have to do some reading/coursework before proceeding efficiently. Here is a very good online-course that covers predictive machine-learning procedures (including the ones listed above) in detail.