Simulating or generating datasets for testing modelling methodologies

Question

I'm interested to know what a good practice is when generating/simulating data for testing or comparing modelling methods. I'm focusing on linear models and measuring prediction accuracy and measuring parameter estimate accuracy.

I'm looking at prediction accuracy (PSE) in terms of cross validated squared prediction error and parameter estimate accuracy in terms of $\|\beta-\hat{\beta}\|$ (MSE).

I have attempted a bunch of simulations using what I believe to be a 'sound' method, but I am not happy with the results. I'm guessing I'm overlooking something. The issue is that adjacent parameter estimates are highly correlated with each other and I cannot figure out why. Another issue is that while I can generate datasets that show a large difference in parameter estimate accuracy, the prediction performance will be very similar. I am not sure why this happens, I would expect the methods with improved MSE to have much improved PSE over the other methods. Here's a typical example: http://ohyur.com/random/typical.png

I will outline what I have attempted.

• Set the true parameter estimates, $\beta$
• Set correlation structure for the predictor variables
• Generate the predictors with specified correlation structure, $X$. I've tried using mvrnorm() in R to generate N(0,1) columns and mvrunif() function to generate U(-1,1) columns. Both with the specified correlation structure.
• Generate the response using $Y=\mu + X\beta+\epsilon$, where $\epsilon$ is $N(0,\sigma$).

To me, that sounds like a reasonably method of generating data for a linear model.

Here is an example of what the generated data might look like: http://ohyur.com/random/data.png ($\mu=3,$ $\beta=(3,2,-3,5,2,-3,5,5)$, $\sigma=15$, $\text{cor}(x_i,x_j)=0.9^{|i-j|}$

Now in my simulations, in each replication I'm generating data according to the above steps. Then fitting some models, estimating the CV PSE and calculating the MSE and storing the results. Here is a boxplot of the results for 3 different modelling methods over 100 replications. http://ohyur.com/random/psemse.png

Now, most interestingly to me, Here is a pairwise plot of the estimated parameters for a full OLS model. Why are the adjacent predictors correlated? http://ohyur.com/random/estimates.png

One thing I'm coverned about is that in the real world, some predictors are more correlated with the response than others, and not just due to effect size. This isn't really accounted for in my setup. But I'm really just after the simplest possible 'sound' method.

Any insight/comments or recommendations on how to properly generate data for testing liner models would be greatly appreciated.

Answer 1

Not an answer, but more of an extended comment:

Regarding your last figure: your predictors are random and correlated, so why wouldn't you expect the corresponding estimators to be correlated?

A possible flaw with your simulation approach is the fact that your predictors are random. In ordinary linear models (the setting that you seem to be interested in), they are assumed to be fixed. When you evalute your methods for random predictors, you are actually evaluating them in an errors-in-variables setting, where the OLS estimator is neither unbiased nor consistent! Thus the results may be misleading... Not sure if this explains the MSE-PSE-inconsistency though.