Here's the scenario: I'm estimating the effect of a treatment on a count outcome. I'm using covariate balancing propensity scores with weighting by the odds to balance my treated and control group, then using a weighted negative binomial regression of my outcome on treatment to estimate the treatment effect (ATT). On top of this, I have missing covariate data, so I've had to use multiple imputation to account for this. I'm using the technique of performing the entire effect estimation in each imputation rather than averaging my imputed propensity scores and performing one effect estimation (I've researched the other method and decide I prefer the one I'm using).
My problem is with standard error estimation. Stata is good at combining multiply imputed results using Rubin's rules, and is good at generating standard errors that account for survey weights (e.g., propensity score weights). The problem, though, is that I don't trust Stata's estimates of the standard errors because simulations have shown that empirical standard deviations of propensity score weighted effect estimators are much smaller than calculated standard errors from WLS (including those with Stata's robust SE adjustment). Bootstrapped standard errors/p-values/confidence intervals seem to have shown promise in this context, so I want to generate them.
My plan was to resample from my data set and run my analysis, and make a list of my treatment effect estimates, then use that list to calculate a standard error and confidence interval for my observed treatment effect.
- Does this seem like a reasonable approach or have I missed steps?
- When resampling, should I resample randomly from my entire sample, or resample my control and treated groups separately? I'm afraid that if I resample from my entire sample, I may end up with some bootstrap samples that have very few treated units. Does it make sense to stratify on the treatment variable before resampling? If not the treatment variable, does it make sense to stratify on some other grouping variable (e.g., cohort)?