8
$\begingroup$

(fairly long post, sorry. It includes lots of background info, so feel free to skip to the question at the bottom.)

Intro: I am working on a project where we are trying to identify the effect of a binary endogenous variable, $x_1$, on a continuous outcome, $y$. We have come up with an instrument, $z_1$, that we strongly believe to be as-if-randomly assigned.

Data: The data itself is in a panel structure with about 34,000 observations spread across 1000 units and about 56 time periods. $x_1$ takes on a value of 1 for about 700 (2 %) of the observations, and $z_1$ does so for about 3000 (9 %). 111 (0.33 %) observations score a 1 on both $z_1$ and on $x_1$, and it is twice as likely for an observation to score a 1 on $x_1$ if it also scores a 1 on $z_1$.

Estimation: We estimate the following 2SLS model through Stata’s ivreg2-procedure:

$$x_1 = \pi_0 + \pi_1z_1 + \mathbf{Z}\mathbf{\pi} + v$$ $$y = \beta_0 + \beta_1 x_1^* + \mathbf{Z}\mathbf{\beta} + u$$

Where $Z$ is a vector of other exogenous variables, $x_1^*$ is the predicted value of $x_1$ from the first stage, and $u$ and $v$ are error terms.

Results: Everything seems to be working well; the estimate of $\pi_1$ is highly significant in the first stage and the estimate of $\beta_1$ is highly significant in the second stage. All signs are as expected, including the ones for the other exogenous variables. The problem is, however, that the estimate of $\beta_1$ – the coefficient of interest – is implausible large (or, at least, according to the way we’ve been interpreting it).

$y$ ranges from about 2 to about 26 with mean and median of 17, but the estimate of $\beta_1$ ranges from 30 to 40 (depending on specification)!

Weak IV: Our first thought was that this was due to the instrument being too weak; that is, not correlated very much with the endogenous variable, but this does not really seem to be the case. To inspect the weakness of the instrument, we use Finlay, Magnusson, and Schaffer’s weakiv-package, as it provides tests that are robust to violations of the $i.i.d.$ assumption (which is relevant here, given that we have panel data and cluster our SE’s at the unit level).

According to their AR-test, the lower bound of the 95 % confidence interval for the second-stage coefficient is between 16 and 29 (again depending on specification). Rejection probability is practically 1 for all values anywhere close to zero.

Influential observations: We’ve tried estimating the model with each unit removed individually, with each observation removed individually, and with clusters of units removed. No real change.

Proposed solution: Someone proposed that we shouldn’t summarize the estimated effect of the instrumented $x_1$ in its original metric (0-1), but in the metric of its predicted version. $x_1^*$ ranges from -0.01 to 0.1 with mean and median of about 0.02 and an SD of about 0.018. If we were to summarize the estimated effect of $x_1$ by, say, a one SD increase in $x_1^*$, that would be $0.018*30 = 0.54$ (other specifications give almost identical results). This would be way more reasonable (yet still substantial). Seems like the perfect solution. Except I’ve never seen anybody do that; everybody just appears to interpret the second-stage coefficient using the metric of the original endogenous variable.

Question: In an IV-model, is it correct to summarize the estimated effect (the LATE, really) of an increase in the endogenous variable by using the metric of the predicted version of it? In our case, that metric is predicted probability.

Note: We use 2SLS even though we have a binary endogenous variable (making the first stage an LPM). This follows Angrist & Krueger (2001): “Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments”) We’ve also tried the three-stage procedure used in Adams, Almeida, & Ferreira (2009): “Understanding the relationship between founder–CEOs and firm performance”. The latter approach, which consists of a probit model followed by 2SLS, yields smaller and more sensible coefficients, but they are still very large if interpreted in the 0-1 metric (about 9-10). We get the same results with manual calculations as we do with the probit-2sls-option in Cerulli’s ivtreatreg.

$\endgroup$
  • $\begingroup$ Have you tried etregress/treatreg? $\endgroup$ – Dimitriy V. Masterov Nov 5 '15 at 2:22
  • $\begingroup$ Hi Dimitriy, thanks for the response! I've tried etregress now, and it gives somewhat similar results. However, reading the Stata manual and Wooldridge (2002): "Econometric analysis of cross section and panel data" I get the impression that this sort of treatment-regression model assumes ignorability of treatment. That is, conditional on the observed variables, whether a unit is treated or not is independent of its (potential) outcome under both treatment and control. $\endgroup$ – Bertel Nov 7 '15 at 1:35
  • $\begingroup$ (cont.) In our data, we can't really sustain this assumption; we merely have a source of random variation in $x$. Therefore, IV seems the appropriate option. If I have the assumptions right, anyway. $\endgroup$ – Bertel Nov 7 '15 at 1:39
  • $\begingroup$ It would be really helpful, to have some graphs, e.g. scatterplots or kernel density plots of the raw varables and the residuals etc.. Remember that plim $\hat{\beta}_1 = \beta_1 + \frac{Cov(z_1,u)}{Cov(z_1,x_1)}$, even a small correlation between the instrument and the error term can cause a strong inconsistent estimate of $\beta_1$! $\endgroup$ – Arne Jonas Warnke Sep 4 '16 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.