Here's the scenario: I'm estimating the effect of a treatment on a count outcome. I'm using covariate balancing propensity scores with weighting by the odds to balance my treated and control group, then using a weighted negative binomial regression of my outcome on treatment to estimate the treatment effect (ATT). On top of this, I have missing covariate data, so I've had to use multiple imputation to account for this. I'm using the technique of performing the entire effect estimation in each imputation rather than averaging my imputed propensity scores and performing one effect estimation (I've researched the other method and decided I prefer the one I'm using).

My problem is with standard error estimation. Stata is good at combining multiply imputed results using Rubin's rules, and is good at generating standard errors that account for survey weights (e.g., propensity score weights). The problem, though, is that I don't trust Stata's estimates of the standard errors because simulations have shown that empirical standard deviations of propensity score weighted effect estimators are much smaller than calculated standard errors from WLS (including those with Stata's robust SE adjustment). Bootstrapped standard errors/p-values/confidence intervals seem to have shown promise in this context, so I want to generate them.

My plan was to resample from my data set and run my analysis, and make a list of my treatment effect estimates, then use that list to calculate a standard error and confidence interval for my observed treatment effect.

  1. Does this seem like a reasonable approach or have I missed steps?
  2. When resampling, should I resample randomly from my entire sample, or resample my control and treated groups separately? I'm afraid that if I resample from my entire sample, I may end up with some bootstrap samples that have very few treated units. Does it make sense to stratify the treatment variable before resampling? If not the treatment variable, does it make sense to stratify on some other grouping variable (e.g., cohort)?

Thank you.

  • $\begingroup$ Perhaps some of the discussion in stats.stackexchange.com/questions/35955/… may help? Not an area with which I am too familiar so difficult to be more helpful. $\endgroup$
    – mdewey
    Nov 27, 2016 at 12:16
  • $\begingroup$ That's a good thread, thank you. They don't talk about bootstrapping unfortunately, but they do talk about estimating SEs without it (using the sandwich estimator) in a context similar to mine. $\endgroup$
    – Noah
    Nov 27, 2016 at 16:29

1 Answer 1


The added variability due to imputation cannot be quantified mathematically. A bootstrap approach would be a way to estimate this variability. If there is a reason why the imputation variability differs for the control group vs the treatment group, a stratified bootstrap would make sense. In any case, it seems to be safe to take this approach. If the two variances are equal that should show in the stratified estimates.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.