Correct way of testing machine learning against random data

I am using a genetic algorithm to search a very complex hypothesis space. Now I want to estimate how much overfitting I can expect in the final resulting hypothesis. The final model will be used for predicting N output variables from M predictors.

One simple test I can do is to replace the predictors with M random variables and then run the same algorithm and compare the final results. This let's me know how much overfitting I can expect. However because I have multiple input variables (predictors) the quality of the final result might also be influenced by the (unknown) underlying distribution and the (also unknown) covariance of these variables.

This leads to the question how I could produce a set of random input variables, which have the same distribution and covariance as the variables I am testing against. One simple solution I could come up with, would be, to just "shuffle" the sets of predictors around. I.e. let

Y_i=(y_{i,1},y_{i,2},...,y_{i,N})


be the variables I want to predict and

X_i=(x_{i,1},x_{i,2},...,x_{i,M})


are the variables I am using for prediction. Then I would use

X'_i=X_j


for a random and unique j for each i and then use the X'_i to determine the Y_i.

Would this work to produce random predictors from the same distribution that can be used for estimating the overfitting? Or are there other effects I need to take into account when doing this kind of test?

• Do you have multiple target variables? Confused by the notation y_{i,1} – B_Miner Nov 3 '11 at 13:27