I am using a genetic algorithm to search a very complex hypothesis space. Now I want to estimate how much overfitting I can expect in the final resulting hypothesis. The final model will be used for predicting
N output variables from
One simple test I can do is to replace the predictors with
M random variables and then run the same algorithm and compare the final results. This let's me know how much overfitting I can expect. However because I have multiple input variables (predictors) the quality of the final result might also be influenced by the (unknown) underlying distribution and the (also unknown) covariance of these variables.
This leads to the question how I could produce a set of random input variables, which have the same distribution and covariance as the variables I am testing against. One simple solution I could come up with, would be, to just "shuffle" the sets of predictors around. I.e. let
be the variables I want to predict and
are the variables I am using for prediction. Then I would use
for a random and unique
j for each
i and then use the
X'_i to determine the
Would this work to produce random predictors from the same distribution that can be used for estimating the overfitting? Or are there other effects I need to take into account when doing this kind of test?