I've set up a measurement system that allows me to measure thousands of different elements, all of which have potential to be used in a regression model. My aim is to create a model that can predict a target (dependent) variable. Making the thousands of measurements is cheap. What is expensive is measuring the target variable, and our budget is blown measuring over 80 specimens.
When it comes to variable selection for the predictive model, I have used both a forward and step-wise selection methods, but then found using a best-subsets method (regsubsets in R) produced even better models (higher adjusted R2 and better Mallow's Cp). However, all of these suggest models that are between 13 and 17 variables, and all the tests suggest that each variable is significant (usually p < .001) and the adjusted R2 is up around 0.95. I am quite dubious about this many, with the potential for overfitting. Given the high number of potential independent variables, I suspect that some of the later ones in the model are no better than chance.
To overcome this potential, I previously doubled the number of IVs, populating the second half with randomly generated numbers (entitled "random IVs" for the remainder of the question). I ensured that each subset had at least 25% from the random IVs group, and worked through all IVs selecting until one random IV appeared in the model. My reasoning behind this ad-hock method is that any IV in a truly predictive model should do better than the best random variable, and therefore despite the low p-value, is not significant. The resulting models used 6-7 IVs, with adjusted R2 values around .85 to 0.9.
My question is therefore two-fold. Firstly, is there a better, formalised test or method to prevent this type of overfitting (specifically associated with large numbers of potential IVs to be tested)? Secondly, if this ad-hock method is appropriate, is there a real limit on the number of random variables that is appropriate? (On this second, I can see that using thousands of random variables may effectively be a very high hurdle to jump.)