Instrument Validity Is Not Empirically Testable

Question

BACKGROUND

Suppose we want to use a random sample to estimate $\beta$ in the following regression model: $$Y_i=\alpha+\beta X_i+\epsilon_i.$$ The OLS estimator of $\beta$ is inconsistent if the explanatory variable $X_i$ is correlated with the error term $\epsilon_i$. One solution to this problem is the so-called instrumental variables (IV) solution. The idea behind the IV solution is to isolate variation in $X_i$ that is unrelated to $\epsilon_i$ by means of a so-called instrument $Z_i$. The resulting IV estimator of $\beta$ is consistent if the IV is valid. In the traditional IV literature, an instrument is valid under the following three conditions:

1. $Z_i$ is as good as randomly assigned.
2. $Z_i$ satisfies the exclusion restriction, i.e. it affects $Y_i$ only through its effect on $X_i$.
3. $Z_i$ affects the endogenous regressor.

(The so-called conditional IV estimator is valid if the instrument satisfies (1)-(3) conditioning on some appropriate set of control variables.)

It is also often said in the traditional IV literature that (3) is testable or empirically verifiable. I believe it is not. I believe that (3) can only be empirically verified if we assume (1) and then test the null hypothesis that $\delta=0$ in the following so-called first stage estimating equation: $$X_i=\gamma+\delta Z_i+\eta_i.$$

For if (1) does not hold, then any estimated relationship between $X_i$ and $Z_i$ that is captured by OLS on the first stage equation is spurious. The figures I have in mind are those in Figure 9 on page 39 in a recent article by Imbens (2019). Indeed, Imbens (2019) notes that an instrument is valid only if there is no unmeasured confounder that affects both the instrument and either the endogenous variable $X_i$ or dependent variable $Y_i$, see Figure 9(c) and 9(d) which represents two different cases where (1) does not hold although both (2) and (3) holds. Here is a relevant quote from Imbens (2019):

The second key assumption is that there is no unmeasured confounder that affects the instrument and either the endogenous variable $X$ or the outcome $Y$.

For the case in Figure 9(c), I argue that one cannot test (3) in the first stage since there is an unmeasured confounder that affects $Z_i$ and $X_i$. (For the case in Figure 9(d), one can test (3), but the instrument is invalid since the reduced form relationship cannot be identified, i.e. the relationship between $Z_i$ and $Y_i$.

SIMPLE THEORETICAL ARGUMENT

Suppose that there is an unmeasured confounder $U_i$ such that $U_i$ affects $X_i$ and also affects $Z_i$. Then (1) is not satisfied. Furthermore, if the effect of $U_i$ on $X_i$ is linear with coefficient $\xi$, the first stage estimator takes the form $$\frac{Cov(X_i,Z_i)}{V(Z_i)}=\delta+\xi\frac{Cov(U_i,Z_i)}{V(Z_i)}\neq\delta.$$ Thus, even with an infinite sample, we cannot use the first stage to assess whether $\delta\neq0$, without assuming that $U_i$ does not affect $Z_i$ (which cannot be justified using data only). Hence, instrument validity is not empirically testable.

QUESTION

Am I right, that in this context, (3) is not testable if we do not assume (1), but that we can empirically test (3) given (1)? Hence, the relevance condition (3) is not empirically testable since it requires that we assume the empirically non-verifiable and non-testable condition (1) (e.g. that variation in $Z_i$ and $X_i$ is not generated by some third variable $U_i$ that affects both $Z_i$ and $X_i$).

Answer 1

You are right about the first stage, but as far as I can tell, it is not necessary for the first stage to recover the causal effect of $Z$ on $X$ for IV to work. Correlation seems to be enough for the first stage. I'm confused that Imbens says otherwise, is that shown somewhere in the paper?

Here is the case depicted in panel c of Figure 9 in the Imbens paper when everything is linear.
$$Y = \beta_0 + \beta_1 X + U_1$$. $U_1$ is unobserved and $E[U_1|X] \neq E[U_1]$ so OLS does not recover $\beta_1$.

Suppose further $$X = \delta_0 + \delta_1 Z + U_2$$ is the true causal model for $X$ and $E[U_2|Z] \neq E[U_2]$, so OLS doesn't recover $\gamma_1$ either, but $E[U_1|Z] = E[U_1]$. We can always write down the linear projection $$X = \gamma_0+ \gamma_1 Z + V$$ where $cov(V, Z) = 0$ by construction, and $\gamma_1$ is what the regression of $X$ on $Z$ will estimate ($\delta_1 \neq \gamma_1$)

What happens if we do IV? The IV estimand is $$\beta_1^{IV} = \frac{cov(Y, Z)}{cov(X, Z)} = \beta_1 \frac{cov(X, Z)}{cov(X,Z)} + \frac{cov(U_1, Z)}{cov(X,Z)} = \beta_1$$ because $cov(U_1, Z) = 0$.

The first stage coefficient $\frac{cov(X,Z)}{var(Z)}$ is indeed not$\delta_1$, it is $\gamma_1$ but that seems to be irrelevant, as long as $Z$ does not correlate with $U_1$, satisfies the exclusion restrictions, and has a first stage ($cov(X,Z) \neq 0$).