0
$\begingroup$

I have a multinomial logit model with three outcome categories and a variety of (primarily binary) independent variables.

How would I go about bootstrapping the average treatment effect and its confidence interval of a binary variable?

Right now my procedure is as follows:

After running the full model as a multinomial logit I

  1. Sample with replacement to create temporary data of same length as original dataset.
  2. Reestimate the model using temporary data
  3. set set indep. variable to 0 in temporary data and predict probabilities of outcomes (using reestimated model)
  4. set set indep. variable to 1 in temporary data and predict probabilities of outcomes (using reestimated model)
  5. Take the mean of the average difference in predicted probabilities
  6. Save this mean difference in pred probabilities
  7. Repeat this process a lot of times
  8. Take mean of all the mean differences pred. probabilities for average treatment effect, Take 0.25 and 0.975 quantile for a 95% confidence interval around the mean.

The issue I am having right now is, that this seems to create very large confidence intervals. This has created a situation where a lot of the variables that are significant in the regression output no longer have an effect that is differentiable from 0.

Two questions:

  1. Is this bootstrap procedure sensible?
  2. If so, what explains the difference between the regression outputs and the bootstrap results?
$\endgroup$
2
$\begingroup$

I've done a very similar analysis using bootstrapping to find the ATE on clinical data.

  1. Your procedure seems sensible, it's just nonparametric bootstrapping, and is what I did. The percentile bootstrap CI intervals do get used, but you could look into fancier types of bootstrap confidence intervals as well.
  2. It could be that your logit model overfit/is not close to the "true model", therefore its CIs are too small. Especially, a big thing to look out for is positivity issues. Suppose your binary covariates are $W_1, W_2, W_3$ respectively, and for $(W_1, W_2, W_3) = (0,1,0)$ you only have 2 observations in this category. They're highly influential, being the only points in this category, but would likely be excluded in the bootstrapped data. This may change your estimated ATE by a lot.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.