7
$\begingroup$

My supervisor and I have run a randomized experiment in a developing country. Due to administrative problems there we unfortunately have the problem of non-response. This non-response is also not random because of a flaw in the experiment that introduced self selection. Hence we cannot hope to credibly sell a missing at random or similar assumption. What was suggested to us by a colleague is to set-identify the average treatment effect instead. In particular, two papers were pointed out:

Manski, 1990, "Nonparametric Bounds on Treatment Effects" and Lee, 2009, "Estimating Sharp Bounds on Treatment Effects".

Since I am not familiar with this literature, I hope to get some guidance. I cannot quite understand the difference between the widths of these bounds, i.e. when will they be informative? I would be happy to have a hint or explanation as concerns the mathematical statistics behind these widths. As far as I see it, I should use the bounds by Lee (2009) but I'm not sure if I can a bound on the average treatment effect for all individuals in the study or for a sub-group.

$\endgroup$
5
$\begingroup$

Let $R_i$ be a dummy which equals one for respondents and zero for non-respondents, $Y_i$ the outcome and $D_i$ the treatment variable from your randomized experiment. You cannot observe the counterfactual quantities that you want to compare, i.e. $E[Y_{i1}|R_i = 0|D_i = 1]$ and $E[Y_{i0}|R_i = 0, D_i = 0]$, due to non-response but you know their probability weights from the data and you can use the fact that $Y_i$ and $R_i$ are bounded between zero and one.

Assuming the worst case scenario in which $E[Y_{i1}|R_i = 0|D_i = 1] = 0$ and $E[Y_{i0}|R_i = 0, D_i = 0] = 1$, the lower Manski bound is given by: $$ \begin{align} B^{L} &= P(R_i = 1|D_i = 1)E(Y_i|D_i = 1, R_i = 1) \newline &- [P(R_i = 1|D_i = 0)E(Y_i|D_i = 0, R_i = 1) + P(R_i = 0|D_i = 0)] \end{align} $$ which is the difference between the outcome of the treated given that they responded, minus the outcome of the non-treated given that they responded plus the probability that non-treated individuals did not respond.

In the same spirit, assuming the best case scenario in which $E[Y_{i1}|R_i = 0|D_i = 1] = 1$ and $E[Y_{i0}|R_i = 0, D_i = 0] = 0$, the upper Manski bound is given by: $$ \begin{align} B^{U} &= P(R_i = 1|D_i = 1)E(Y_i|D_i = 1, R_i = 1) + P(R_i = 0 | D_i = 1) \newline &- P(R_i = 1|D_i = 0)E(Y_i|D_i = 0, R_i = 1) \end{align} $$

Naturally the average treatment effect will lie in between those extreme cases. The width of the Manski bounds is simply the difference between the upper and lower bound: $$\text{width} = P(R_i = 0|D_i = 1) + P(R_i = 0|D_i = 0)$$ This width is determined by the probabilities of selection into treatment given the response behavior, i.e. more non-response increases the width. The bounds were to be informative if both upper and lower bound could jointly lie to either side of zero. It turns out that they never can, so you cannot sign your treatment effect.
At best you can get an idea about the range of your effect but in general Manski bounds are uninformative. You cannot make these bounds any smaller without imposing additional assumptions.

Lee's (2009) bounds on the other hand are narrower but they only bound a specific treatment effect. I'm not deriving the width of the bounds again now because they are obtained in the same way as for the Manski bounds. When you subtract the lower bound from the upper bound you get: $$\text{width} = \frac{P(R_i = 1|D_i = 1) - P(R_i = 1|D_i = 0)}{P(R_i = 1|D_i = 0)}$$ If the difference in response rates across treated and non-treated individuals is small, the bounds will be informative.
The crucial assumption is that this difference in response rates is not because of a difference between these two groups but because treatment has an effect on response. In your particular case this assumption does not seem to hold because as far as I see it you lost a batch of responses or some administrative issue held you back. An example for when this assumption holds is if people are angry if they didn't get the treatment and therefore do not respond, or people who got the treatment do not care anymore about responding because they have gotten what they wanted.

Even if you can credibly make this assumption, Lee's bounds cannot distinguish between people who always respond and those who respond because they received the treatment. Lee calls those always respondents and response compliers. So what you are bounding with this is the average treatment effect of these two subpopulations. Whether this is what you want really depends on your application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.