IPTW ATE Significance Test

Question

Does anyone have any experience using propensity weighting schemes such as IPTW (Inverse Probability of Treatment Weighting) estimation? I have a model that uses IPTW to estimate the Average Treatment Effect from observational data, but I'm not sure how to interpret the results or test the significance.

It's a (sorta) controlled experiment, so we actually have access to the real propensity scores. The procedure goes like this:

1. Assign individuals to treatment with some probability

2. Weight the outcomes for each individual based on the inverse of the probability of treatment (propensity score)

3. Compare the mean outcomes of treatment and control groups to form ATE

Now I have the ATE, but I don't know how to test my null (i.e. P(ATE <= 0)). Does anyone have any idea how this is done usually?

Answer 1

Typically, you run a weighted regression model of the outcome on the treatment, and use robust standard errors to estimate the confidence interval and p-value. See Harder, Stuart, & Anthony (2010) for an example and example R code. The coefficient on the treatment in the regression is the effect estimate (i.e., the ATE), and the intercept is the counterfactual mean under the reference category (i.e., the control group if your treatment variable is dummy-coded with 1 as treated).

The interpretation of the ATE is the average difference between the potential outcomes under treatment and the potential outcomes under control in the population from which your sample was drawn, which is the same interpretation as it would be in a completely randomized experiment.

One final note is that if you have a model that perfectly explains the propensity scores (e.g., males get a probability of treatment of .5, females get a probability of treatment of .7), it ends up being better to estimate the propensity scores in your sample using the correct model than it is to use the true propensity scores.