3
$\begingroup$

I'm interested to know what a good practice is when generating/simulating data for testing or comparing modelling methods. I'm focusing on linear models and measuring prediction accuracy and measuring parameter estimate accuracy.

I'm looking at prediction accuracy (PSE) in terms of cross validated squared prediction error and parameter estimate accuracy in terms of $\|\beta-\hat{\beta}\|$ (MSE).

I have attempted a bunch of simulations using what I believe to be a 'sound' method, but I am not happy with the results. I'm guessing I'm overlooking something. The issue is that adjacent parameter estimates are highly correlated with each other and I cannot figure out why. Another issue is that while I can generate datasets that show a large difference in parameter estimate accuracy, the prediction performance will be very similar. I am not sure why this happens, I would expect the methods with improved MSE to have much improved PSE over the other methods. Here's a typical example: http://ohyur.com/random/typical.png

I will outline what I have attempted.

  • Set the true parameter estimates, $\beta$
  • Set correlation structure for the predictor variables
  • Generate the predictors with specified correlation structure, $X$. I've tried using mvrnorm() in R to generate N(0,1) columns and mvrunif() function to generate U(-1,1) columns. Both with the specified correlation structure.
  • Generate the response using $Y=\mu + X\beta+\epsilon$, where $\epsilon$ is $N(0,\sigma$).

To me, that sounds like a reasonably method of generating data for a linear model.

Here is an example of what the generated data might look like: http://ohyur.com/random/data.png ($\mu=3,$ $\beta=(3,2,-3,5,2,-3,5,5)$, $\sigma=15$, $\text{cor}(x_i,x_j)=0.9^{|i-j|}$

Now in my simulations, in each replication I'm generating data according to the above steps. Then fitting some models, estimating the CV PSE and calculating the MSE and storing the results. Here is a boxplot of the results for 3 different modelling methods over 100 replications. http://ohyur.com/random/psemse.png

Now, most interestingly to me, Here is a pairwise plot of the estimated parameters for a full OLS model. Why are the adjacent predictors correlated? http://ohyur.com/random/estimates.png

One thing I'm coverned about is that in the real world, some predictors are more correlated with the response than others, and not just due to effect size. This isn't really accounted for in my setup. But I'm really just after the simplest possible 'sound' method.

Any insight/comments or recommendations on how to properly generate data for testing liner models would be greatly appreciated.

|cite|improve this question
$\endgroup$
2
$\begingroup$

Not an answer, but more of an extended comment:

Regarding your last figure: your predictors are random and correlated, so why wouldn't you expect the corresponding estimators to be correlated?

A possible flaw with your simulation approach is the fact that your predictors are random. In ordinary linear models (the setting that you seem to be interested in), they are assumed to be fixed. When you evalute your methods for random predictors, you are actually evaluating them in an errors-in-variables setting, where the OLS estimator is neither unbiased nor consistent! Thus the results may be misleading... Not sure if this explains the MSE-PSE-inconsistency though.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Interesting. But why are only adjacent estimates correlated? I'll check out this errors in variables setting to learn more about it. $\endgroup$ – dcl May 21 '12 at 8:23
  • $\begingroup$ The adjacent estimators should at least have a higher correlation since the corresponding predictors have higher correlations... Have you tried computing the hat matrix for a particular simulated sample or estimating the expected hat matrix? That contains the covariance strucutre of the estimators; I guess that it turns out the only highly correlated predictors get moderately correlated estimators. $\endgroup$ – MånsT May 21 '12 at 9:12
  • $\begingroup$ I will try checking out the hat matrix now. I tried an example with fairly uncorrelated predictors ($\rho_{ij}=0.25^{|i-j|}$) and got the following: The data: ohyur.com/random/lowdata.png The estimates: ohyur.com/random/lowestimates.png . So even while the pairwise correlations in the other data are much stronger than these, it is still only the adjacent predictors that are correlated. $\endgroup$ – dcl May 22 '12 at 2:59
  • $\begingroup$ In this context, how would I interpret the n by n hat matrix? This is what it looks like for a single rep with 120 observations ohyur.com/random/hat.png they're fairly similar over the 100 reps $\endgroup$ – dcl May 22 '12 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.