I need to check the coverage of a confidence interval, but I don't know which one of the following approaches I should use.

Approach 1:

  1. Estimate the regression parameter $\theta$
  2. Use the sandwich estimator to estimate the variance $\hat{\sigma}$
  3. Create the confidence interval (assuming asymptotic normality)

Repeat it N times and get the proportion of times that the true $\theta$ falls inside the confidence interval.

Approach 2:

  1. Estimate the regression parameter $\theta$ for N simulations
  2. Take the standard error of $\hat{\theta}$, which returns a single number
  3. Create N confidence intervals (assuming asymptotic normality) based on the estimates of $\theta$ and the standard error of $\hat{\theta}$
  4. Get the proportion of times that the true $\theta$ falls inside the confidence interval

In summary, the first one we check if the true $\theta$ falls inside the confidence interval at each simulation (hence, we use the sandwich estimator for the variance). In the second one, we check only after the N simulations have been done, so that the standard error of the estimates is used to create the interval.\

They will create different confidence interval with different coverage. Which one should I use? I have seem people using both.

  • 1
    $\begingroup$ As a side note, I see that both the sandwich and empirical variance estimates will be asymptotically equivalent, but they differ in finite samples. So I don't know which one to use. Thanks for the help. $\endgroup$ – bnh May 13 '15 at 14:12

Approach 1 is the correct way to assess the coverage probability of an estimator. An estimator along with its variance estimate (which is also an estimator) are used to create a 95% CI independent of any knowledge of what future realizations of these data are. If you create N=5000 simulated realizations from the data and take the 2.5 and 97.5 quantiles, then you'll get 95% coverage each time... however, this has no connection to statistics. Each of those N=5000 realizations corresponds to one instance of the statistical experiment. The point of statistics is using the information from N=1 experiment to infer what the estimator will tend to be in the future N=4999 experiments. The estimator must be a function of a single dataset.

The confusing bit may come from bootstrapping where for each N=1 experiment, you bootstrap simulated datasets N*=1000 based on the empirical distribution of the N=1 dataset. These empirical distributions are different for each of the N=5000 simulated realizations, however.

| cite | improve this answer | |

The correct interpretation of a confidence interval of say 90% is that if you would fit models on multiple simulations or samples, then in 90% of the time the true value will be in the confidence interval. For more information see: http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and_interpretation

Confidence intervals are underestimated ;-)

| cite | improve this answer | |
  • $\begingroup$ What, then, is the answer to the question? Approach, 1, approach 2, or neither? And why? $\endgroup$ – whuber May 13 '15 at 13:29
  • 1
    $\begingroup$ Thanks for the answer, but as @whuber pointed out, I can't see how this relates to the question. Could you elaborate a little more, please? Or perhaps let me know if the question is not clear so I can rewrite it. $\endgroup$ – bnh May 13 '15 at 14:09
  • $\begingroup$ My intention was to support the first option, my answer is in line with the summary sentence: "In summary, the first one we check if the true θ falls inside the confidence interval at each simulation" This option narrowly follows the definition in the given reference. To construct the confidence intervals one would need an standard error of the estimates; this could be on the basis of a sandwich estimator. but I would consider this a special case.The message I was trying to convey is that the 90% should be seen as in the context of (normally) hypothetical resampling, in line with option 1. $\endgroup$ – spdrnl May 13 '15 at 15:21
  • $\begingroup$ Thank you very much for elaborating more on your response, @spdrnl. It is much more clear now. Thank you! $\endgroup$ – bnh May 14 '15 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.