4
$\begingroup$

I want to check that the endogeneity of a treatment variable gets eliminated by conditioning on a set of control variables, but I am not entirely sure on how to proceed. I have tried one approach and I would welcome your thoughts.

I have a cross-sectional dataset with 200K+ individuals, households being the sampling units. I want to estimate the causal effect of some dichotomous treatment $X$ on a continuous outcome $Y$. My basic model is as follows:

$$y_{h,i} = \beta_0 + \beta x_{h,i} + \epsilon_{h,i},$$

where $\epsilon_{h,i}=\eta_h+\upsilon_{h,i}$, $\eta_h$ being a household (fixed or random) effect, and $\upsilon_{h,i}$ being idiosyncratic error.

The problem is that $X$ is endogenous, so OLS is inconsistent. I need to employ other estimators for causal inference, and in my case matching estimators are the most appropriate (I don't have a good instrument for $X$ and am not in a panel setting so as to use DiD). I have a large sample size so I can afford exact matching. I use the R package MatchIt (method="exact") to do the matching.

Call $\mathbf{Z}$ the variables I use to exactly match treated and untreated individuals. The matching procedure returns a vector of strictly positive weights, $W$, such that, if $x_{h,i}=1$, $w_{h,i}=1$; otherwise, given $\mathbf{Z}$, $w_{h,i}=n_{\mathbf{Z},t}/(n_{\mathbf{Z}}-n_{\mathbf{Z},t})$, i.e. the weight is increasing in the number of matched treated individuals and decreasing in the total number of individuals equal in $\mathbf{Z}$. The point of these weights is to run WLS, using the following weighted version of the above model:

$$w_{h,i}y_{h,i} = w_{h,i}(\beta_0 + \beta x_{h,i} + \epsilon_{h,i}).$$

Provided that $\mathbf{Z}$ suffices to conditionally break the endogeneity of $X$ (through the weight adjustments $W$), my understanding is that WLS returns an unbiased estimate of $\beta$, which is interpreted as the average treatment effect on the treated.

Right, but does the $\mathbf{Z}$ I chose do the job? I want to test this. I want evidence that matching individuals on $\mathbf{Z}$ breaks the endogeneity of $X$ and makes my set-up random-like, so I can make causal inference. Essentially, I want to test that $E(\epsilon_{h,i}|X,\mathbf{Z})=E(W^T\mathbf{\epsilon}|W^TX)=0$ (please forgive the slightly informal notation), knowing that $E(\epsilon_{h,i}|X)\neq0$ most likely.

For a start, I ran a Hausman test to check that $Cov(\eta_h,X|\mathbf{Z})=0$, i.e. that random effects were preferred to fixed effects in the weighted model. I found this was the case (while $Cov(\eta_h,X)\neq0$, suggesting fixed effects were preferred in the unweighted model). This is a good result, since it shows that $X$ is uncorrelated with part of $\epsilon_{h,i}$ after matching. Yet it is not enough. I also want to test that $Cov(\upsilon_{h,i},X|\mathbf{Z})=0$.

At first I thought I could run a classic test for endogeneity by regressing $X$ on $\mathbf{Z}$ and then adding the residuals as regressors in the weighted model. If the coefficient on these residuals had been non-zero, I would have concluded $X$ was still endogenous, and so that matching on $\mathbf{Z}$ did not fully eliminate endogeneity. However, the premise of this test is that $\mathbf{Z}$ is uncorrelated to $Y$, and I have good reasons to believe the opposite. Thus specified, the endogeneity test won't work.

Hence, instead of regressing $X$ on $\mathbf{Z}$, and knowing I do not have any good instrument for $X$ at all, I thought I could try the following:

$$x_{h,i} = \delta_0 + \delta w_{h,i} + \zeta_h + v_{h,i},$$

where $\zeta_h$ is a household random effect and $v_{h,i}$ idiosyncratic error. I thought I could then test for the endogeneity of $X$ as usual, by adding the residuals of the last model (the fitted values of $v_{h,i}$) as regressors in the weighted model above and examine the coefficient thereon.

I reckon the last is kind of an odd model. Yet $X$ and $W$ are very much correlated ($x_{h,i}=1$ implies $w_{h,i}=1$), so we can explain the former in terms of the latter. Notice also that $W$ is uncorrelated to $Y$, except through $X$: by construction, it only depends on $X$, $n_{\mathbf{Z}}$ and $n_{\mathbf{Z},t}$, the latter two being uncorrelated to $Y$. For this reason, $W$ qualifies as an 'instrument' for $X$. This is why I tried to test for endogeneity using this pseudo-instrumental variable. In this way, I am using the same variable, $W$, both as an instrument (in the $X$-regression) and as weights (in the weighted $Y$-regression). Notice also that, again by construction, $Cov(\zeta_h,W)=0$, hence the choice of random as opposed to fixed effects.

Following this approach, I got results that make some sense. The residuals had a zero coefficient when plugged into the weighted model, suggesting $X$ was no longer endogenous after matching on $\mathbf{Z}$. I then tried to do the matching using smaller set of $\mathbf{Z}$ (call it $\mathbf{Z}'$), which I did not believe to be able to break endogeneity. I used the newly obtained vector of weights (call it $W'$) in place of $W$, and I found the coefficient on residuals to be non-zero, suggesting that $X$ was still endogenous after such 'reduced' matching, as expected.

Now, my question is: does my procedure make sense to you as it does to me? I hope I made clear that my goal is to check that, after matching treated and untreated individuals on some set of controls, treatment assignment becomes exogenous. My procedure is meant to make this check. Do you think it is sound? Does it need some adjustments? Or is it complete nonsense? Is there a better way? I would appreciate some feedback, references, or advice on how to deal with the matter alternatively. Thank you!

$\endgroup$
  • $\begingroup$ Could you set up the front-door adjustment formula? That would involve inserting a measured variable in-between $X$ and $Y,$ but I don't know if you have that data available. $\endgroup$ – Adrian Keister Jul 21 '20 at 13:57
  • $\begingroup$ Not sure what you mean. If you're asking whether I can specify the weighted model so I have unweighted variables alongside weighted ones, I don't believe I can. $\endgroup$ – Stefano Testoni Jul 21 '20 at 14:05
  • $\begingroup$ I'm talking about causal diagrams. You have essentially a confounding variable influencing both $X$ and $Y.$ There are three common approaches to solving the problem: the back-door adjustment formula (the simplest, as it requires no more variables), the instrumental variable approach (which you've ruled out), and the front-door adjustment formula. By "inserting a variable between $X$ and $Y$", I mean that instead of the causal diagram $X\to Y,$ you insert another variable, $T,$ say, like this: $X\to T\to Y.$ Then you can use the front-door adjustment formula. $\endgroup$ – Adrian Keister Jul 21 '20 at 14:29
  • $\begingroup$ Thanks for clarifying, I was unfamiliar with the language you used. I might have such a $T$, which is kind of a sub-outcome of $X$ affecting the main outcome $Y$. I have several potential $T$s, however, and doing this for all of them is cumbersome, and many $T$s are unobserved. Also, there is no guarantee that observed $T$s are not endogenous, so I would have a causality problem again. $\endgroup$ – Stefano Testoni Jul 21 '20 at 14:36
  • $\begingroup$ For front-door, your $T$ would definitely have to be observed, and would, itself, need to be exogenous, not endogenous. I would encourage you to think in terms of causal diagrams. There's an awful lot you can do with them. If you do come up with one, it wouldn't hurt to add it to your question. $\endgroup$ – Adrian Keister Jul 21 '20 at 16:06
1
$\begingroup$

A) Matching relies on exactly the same set of identification conditions as OLS, and differs only in the lack of a requirement to make assumptions about the functional forms relating treatment/confounders to the outcome - if a causal effect is unidentified for OLS, it is unidentified for matching.

B) The basic assumption you're talking about (selection on observables) is untestable. There's nothing you can do with observed data to determine whether it's true or not, because it relies precisely on information that is missing (the potential outcomes under treatment/control).

If you don't have instruments/natural experiments, you can try to look for 'falsification tests' - i.e. outcomes that may be affected by shared confounders, but where you know (or are willing to assume, based on your knowledge of the area) that there can be no causal effect of treatment. If you find an effect of treatment on such an outcome net of controls, it indicates that there is unobserved confounding and your estimates in the main analysis are biased. If not - they may still be biased.

Hope that's helpful - the main point is that you can't determine whether selection on observables holds based on observed data.

$\endgroup$
  • $\begingroup$ I see your point, but even endogeneity is fundamentally untestable -- yet we have a test for endogeneity. Obviously the test is as good as the degree to which the underlying assumptions apply; whether or not they do cannot be determined (and that's what makes endogeneity fundamentally untestable), but if we take them to apply, the result of the test is informative. The same applies to selection on observables. My procedure is meant to test for it. I'd like to know from you guys if you think it's sound. If it is, then testability depends only on whether the underlying assumptions apply. $\endgroup$ – Stefano Testoni Jul 22 '20 at 15:34
0
$\begingroup$

Here are a few suggestions:

  1. Falsification/placebo test. As suggested in the first answer. If you have an alternative outcome that you know is not affected by treatment then you can assess the credibility of your primary results; e.g, if the main outcome is income after treatment then income before treatment is an ideal candidate.
  2. Covariate balance. If you have additional covariates (potential confounders) that you did not use in your matching you can check for balance between treated and non-treated individuals across these. Significant differences between groups makes the results less credible.

Assuming you run a regression with covariates:

  1. Coefficient stability. Does the estimated treatment effect change much when you add additional potential confounders? If yes, results are less credible.
  2. ”Oster test”. Based on the difference between uncontrolled and controlled treatment effect and R-squared you can calculate a bias-adjusted treatment effect. In essence, if including covariates affected the estimated treatment effect very little but led to a much higher R-squared, results are more credible.

I agree with the first answer. There is no statistical test for this. It’s about assessing credibility.

Regarding your matching approach, why are you doing WLS? Based on my (admittedly limited) knowledge of matching, we would typically match one treated individual to one control individual and discard the rest. Not sure what is to be gained from keeping the lot of them?

If you are using matching instead of regression because you believe that your covariates are endogenous I don’t think you’ve avoided the problem. If they are bad controls you should not be controlling for them by matching or regression. However, bad controls may be better than no controls. My point is that since you are doing regression you might as well include the covariates.

Also, I would not rely on the Hausman test to decide whether or not to use fixed effects. With fixed effects for households you are comparing outcomes within each household. The identification strategy is very different from comparing individuals of different households. And if you believe that random effects are fine, then OLS with robust standard errors should also be fine which seems a more standard choice than GLS imho.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.