Comparing simulated and actual data

Say I have a simple linear regression with observed values of y

y = a + bx

I then have 1000 simulations whereby I generated y randomly. (It's not a matter of reshuffling y; I generated y through a separate model.) For each simulation I have independent estimates of b. My goal is to test whether my actual estimate for b differs from my simulated b's. Normally, I would test whether b falls within the 95% confidence interval of the 1000 simulated b's. However, each b estimate has it's own standard error. How do I take this into account?

I saw one publication that gave the number of simulations that have a b +/i 95% CI that overlaps with the 95% CI of the b from the observed data. Is that the standard way of presenting the results? Also, what if I wanted to show the results (x-y relationship) graphically? What would I use for the 95% confidence interval of the slope of the simulations?

Thank you!

While each parameter estimate has a standard error, as you say, what is that standard error a measure of? It's an attempt to estimate the standard deviation of the sampling distribution of the parameter.

But you have that distribution via your simulation. The sample $b$'s from your simulation represent draws from the sampling distribution. The estimates of standard error are redundant. Don't take it into account, do as you suggest under 'normally' - the interval you generate by simulation should behave as a sample from the sampling distribution under the conditions set up in your simulation (personally I'd use a bigger sample, if possible, but that's just me - many people are content with n=1000).

What would I use for the 95% confidence interval of the slope of the simulations?

The obvious (2.5%ile, 97.5%ile) of the simulated slopes would be the typical thing to do.

• Thank you Glen_b for your answer. My regressions are actually mixed models, which complicates things. My goal is to compare the estimates and predictions from my actual data to those from the simulated data by testing whether their CI's overlap. Following the methods on glmm.wikidot, one can calculate prediction intervals taking into account uncertainty of the fixed effects and the random effects. Does my simulation procedure account for the random effect variance? Would a better approach be to generate confidence intervals via bootstrapping using bootMER? – r_e_f Feb 17 '14 at 15:46
• My regressions are actually mixed models, which complicates things -- that's rather an understatement. Why would you leave that critical information out? Indeed, the first sentence of your question directly states 'simple linear regression'. Why would my answer to the question you asked apply to a very different situation? It's not a trivial change in the circumstances and I don't believe I now understand clearly what simulation procedure you're asking about. This is an entirely new question. – Glen_b -Reinstate Monica Feb 17 '14 at 18:54
• I wanted to learn about the general case, first. I wasn't sure whether to follow up the simple case with the more specialized situation of mixed models or whether to ask a new question. I chose the former and it seems I chose poorly. – r_e_f Feb 17 '14 at 19:11
• It's not a problem - just ask a new question. – Glen_b -Reinstate Monica Feb 17 '14 at 19:12