I'm interested to know what a good practice is when generating/simulating data for testing or comparing modelling methods. I'm focusing on linear models and measuring prediction accuracy and measuring parameter estimate accuracy.
I'm looking at prediction accuracy (PSE) in terms of cross validated squared prediction error and parameter estimate accuracy in terms of $\|\beta-\hat{\beta}\|$ (MSE).
I have attempted a bunch of simulations using what I believe to be a 'sound' method, but I am not happy with the results. I'm guessing I'm overlooking something. The issue is that adjacent parameter estimates are highly correlated with each other and I cannot figure out why. Another issue is that while I can generate datasets that show a large difference in parameter estimate accuracy, the prediction performance will be very similar. I am not sure why this happens, I would expect the methods with improved MSE to have much improved PSE over the other methods. Here's a typical example: http://ohyur.com/random/typical.png
I will outline what I have attempted.
- Set the true parameter estimates, $\beta$
- Set correlation structure for the predictor variables
- Generate the predictors with specified correlation structure, $X$. I've tried using mvrnorm() in R to generate N(0,1) columns and mvrunif() function to generate U(-1,1) columns. Both with the specified correlation structure.
- Generate the response using $Y=\mu + X\beta+\epsilon$, where $\epsilon$ is $N(0,\sigma$).
To me, that sounds like a reasonably method of generating data for a linear model.
Here is an example of what the generated data might look like: http://ohyur.com/random/data.png ($\mu=3,$ $\beta=(3,2,-3,5,2,-3,5,5)$, $\sigma=15$, $\text{cor}(x_i,x_j)=0.9^{|i-j|}$
Now in my simulations, in each replication I'm generating data according to the above steps. Then fitting some models, estimating the CV PSE and calculating the MSE and storing the results. Here is a boxplot of the results for 3 different modelling methods over 100 replications. http://ohyur.com/random/psemse.png
Now, most interestingly to me, Here is a pairwise plot of the estimated parameters for a full OLS model. Why are the adjacent predictors correlated? http://ohyur.com/random/estimates.png
One thing I'm coverned about is that in the real world, some predictors are more correlated with the response than others, and not just due to effect size. This isn't really accounted for in my setup. But I'm really just after the simplest possible 'sound' method.
Any insight/comments or recommendations on how to properly generate data for testing liner models would be greatly appreciated.