coverage of confidence interval

Question

I need to check the coverage of a confidence interval, but I don't know which one of the following approaches I should use.

Approach 1:

1. Estimate the regression parameter $\theta$
2. Use the sandwich estimator to estimate the variance $\hat{\sigma}$
3. Create the confidence interval (assuming asymptotic normality)

Repeat it N times and get the proportion of times that the true $\theta$ falls inside the confidence interval.

Approach 2:

1. Estimate the regression parameter $\theta$ for N simulations
2. Take the standard error of $\hat{\theta}$, which returns a single number
3. Create N confidence intervals (assuming asymptotic normality) based on the estimates of $\theta$ and the standard error of $\hat{\theta}$
4. Get the proportion of times that the true $\theta$ falls inside the confidence interval

In summary, the first one we check if the true $\theta$ falls inside the confidence interval at each simulation (hence, we use the sandwich estimator for the variance). In the second one, we check only after the N simulations have been done, so that the standard error of the estimates is used to create the interval.\

They will create different confidence interval with different coverage. Which one should I use? I have seem people using both.

Answer 1

Approach 1 is the correct way to assess the coverage probability of an estimator. An estimator along with its variance estimate (which is also an estimator) are used to create a 95% CI independent of any knowledge of what future realizations of these data are. If you create N=5000 simulated realizations from the data and take the 2.5 and 97.5 quantiles, then you'll get 95% coverage each time... however, this has no connection to statistics. Each of those N=5000 realizations corresponds to one instance of the statistical experiment. The point of statistics is using the information from N=1 experiment to infer what the estimator will tend to be in the future N=4999 experiments. The estimator must be a function of a single dataset.

The confusing bit may come from bootstrapping where for each N=1 experiment, you bootstrap simulated datasets N*=1000 based on the empirical distribution of the N=1 dataset. These empirical distributions are different for each of the N=5000 simulated realizations, however.

Answer 2

The correct interpretation of a confidence interval of say 90% is that if you would fit models on multiple simulations or samples, then in 90% of the time the true value will be in the confidence interval. For more information see: http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and_interpretation

Confidence intervals are underestimated ;-)