3
$\begingroup$

I'd like to raise a controversial point: if you need instrumental variables, your model is wrong.

Basic endogeneity problem and the IV solution

Let us suppose the basic framework of endogeneity and instrumental variables (IV): we want to estimate the regression function $E(y|x,z)$, and choose a linear model of $y$ on $x$ and $z$, where $x$ is endogenous and $z$ is not. Endogeneity is a serious problem, and can stem from many causes: reverse causation, simultaneity, omitted variables, measurement errors correlated to $y$, model misspecification, etc.

$y_i= \beta_0 + \beta_1 x_i + \beta_2 z_i + \varepsilon_i$ (1)

$E(\varepsilon|x) \neq 0$, $E(\varepsilon|z) = 0$

The OLS estimate of the $x$ parameter $\hat{\beta_1}_{OLS}$ is biased and inconsistent because of the endogeneity of $x$. We thus turn to an instrument, say $w$, which should be (ideally well) correlated to $x$ (conditional on the other regressors) and uncorrelated to the error term $\varepsilon$ of the previous model, in practice:

$E(x|w) \neq E(x)$, $E(x|w) \neq E(x|z,w)$, $E(\varepsilon|w) = 0$

The IV method can be seen as replacing $x$ by $\hat x_{z,w}$ in model (1), where $\hat x_{z,w}$ is the prediction of a linear regression of $x$ on $z$ and $w$. OLS on this new model yields a consistent estimate $\hat{\beta_1}_{IV}$ of the parameter of interest $\beta_1$.

Interpretation of IV estimates

Notice that the IV fit necessarily has a lesser predictive power compared to the OLS fit, because by definition OLS has the lowest mean squared error among all linear models. In econometrics this is justified as a trade-off between goodness-of-fit and consistency of the parameter estimation: the main interest lies not in prediction but in the estimated values of certain parameters, usually in linear models derived from economic theory.

My problem is that most econometric models interpret these parameters of interest ($\beta_1$ in model 1), as the partial derivative $\frac{\partial y}{\partial x}$, the mean effect of a variation of $x$ on the variation of $y$. This is obviously wrong, because in order to compute $\hat{\beta_1}_{IV}$ we deliberately ignored the non-linear effect that $x$ has on $y$ through $\varepsilon$, restricting ourselves to the $\beta_1$ component. This is an important point to make, given the high regard in which IV estimates are held by academics and policy makers.

Of course one could argue that the goal of modelization is to approximate reality, and that to capture the isolated linear effect is better than nothing. The fact of the matter, I would argue, is that if modelization and the estimation of $\frac{\partial y}{\partial x}$ are to be taken seriously, they should not be estimated by sweeping under the rug those relationships that are present in the data but do not conform with the model specification.

If you need IV, your model is wrong

If the point of interest is the estimation of the effect of $x$ on $y$ (given the other regressors) and if $x$ is endogenous in the linear model, then the specified linear model is just wrong, because it violates the hypothesis of non-correlation of $\varepsilon$. Using IV seems to amount to just ignoring this problem and return the "pure" linear effect of an erroneous model, which sounds rather insulting to the scientific spirit. At best, it can be seen as a truncated Taylor expansion, which will only be valid close to the mean point of the dataset.

So rather than resorting to IV when confronted with endogeneity, it seems to me one should aim to find a better model to explain the relationship. A great many "universal approximation" models are readily available (kernel regression, lasso, gradient boosting, neural networks...) and can be used as a benchmark in comparison with the prediction power of any specified model. If a model has poor prediction power, it is a poor representation of reality.

If the final aim is to estimate the "full" partial effect of $x$ on $y$, algorithms such as local kernel-weighted regressions can be run at the points of interest, the parameters of which can be interpreted as local partial derivatives.

And if one is stuck with an endogeneity-ridden linear model because of economic theory, then IV estimates will only be gross approximations of the true effects, and conclusions will only be misleading, right?

Any comments, or references to related literature? (sorry for the long post)

$\endgroup$
  • $\begingroup$ If I get you right, you're saying that the error term always moves with $x$, so getting the linear coefficient on $x$ will ignore the correlated effects in the error term. It is true that observational variations in $x$ will change $y$ through $\epsilon$ as well as $x$. The interpretation of the IV however is generally "what would happen if someone went and set some policy that exogenously changed $x$ and only $x$?" Of course, the imposition of such a policy would probably change a whole other set of unobservables (Hawthorne effects, etc), but that's a different problem. $\endgroup$ – generic_user Jul 21 '14 at 12:47
  • $\begingroup$ One more thing: you keep harping on the linearity issue. Economists definitely rely too much on linear models. Two-stage generalized additive models are possible, as described in this under-cited (to my mind) paper: ucl.ac.uk/statistics/research/pdfs/rr309.pdf $\endgroup$ – generic_user Jul 21 '14 at 12:52
  • $\begingroup$ Thank you for the nice reference! About your comment, my present point is not so much the "ceteris paribus" assumption of a policy that exogenously changed $x$ and only $x$ (which is a serious problem on its own, of course), but rather the fact that IV methodology chooses to ignore part of the effects of $x$ itself. $\endgroup$ – jubo Jul 21 '14 at 17:59
3
$\begingroup$

You raise too many issues, and for most of them any answer will probably be seen as being primarily opinion-based, so this post may be quickly closed.

To provide some comments on some of them:

Quote: If a model has poor prediction power, it is a poor representation of reality.

This would be true in the natural sciences, where physical laws appear to not change over the years. But in social sciences, "laws" change over the years or over different socio-economical environments. Example: the econometric models that failed to cope with the Oil crisis of '73 became useless for the crisis and for after the crisis. They were doing a pretty good job up to that point, and even today, they remain pretty adequate models to describe the economy before that crisis. "Reality" is not the same as "future".

Quote: (The IV estimator)...At best, it can be seen as a truncated Taylor expansion, which will only be valid close to the mean point of the dataset.

and

Quote: ... IV estimates will only be gross approximations of the true effects, and conclusions will only be misleading, right?

...which implies that you somehow are certain that the correlation of the regressor with the error term is in most cases "large" and "strong". Is there evidence of such?

Mind you, I am not in love with IV estimation -I am just trying to point to you aspects of your deliberations that could be made more robust.

$\endgroup$
  • $\begingroup$ Thanks. About "Reality" is not the same as "future", I couldn't agree more, and what I meant by "prediction" is just prediction on withheld data that follow the same distribution as the training data. $\endgroup$ – jubo Jul 21 '14 at 4:18
  • $\begingroup$ And about your second point: the reverse point can be made just as easily, IV supposes that the correlation with the error term is irrelevant. $\endgroup$ – jubo Jul 21 '14 at 4:20
  • $\begingroup$ Exactly... which means that the argument against the IV has just been weakened, from "The IV ignores important correlations" to "The IV ignores correlations that could be important" -and there is a very big difference between the two. The point here is that to assess an estimation method one needs to quantify somehow its flaws and merits, or at least rank the estimator based on its relative flaws and merits, against other available (or newly proposed) estimators. And this is not what your post is doing... although it appears that you do have some concrete ideas towards such a direction. $\endgroup$ – Alecos Papadopoulos Jul 21 '14 at 9:46
  • $\begingroup$ Ok, so we agree that IV ignores potentially important relationships in the data, namely the effect of $x$ on $y$ through $\varepsilon$. My point is that IV is typically used when an endogeneity test (eg. Durbin-Wu) says this effect is significant, and then chooses to continue as if this effect had no part in the partial derivative of $y$ on $x$. Thus the vey argument taken to justify the use of IV tends to indicate that the ignored effect is sizeable. $\endgroup$ – jubo Jul 21 '14 at 17:50
  • $\begingroup$ Are you by any chance implying that "statistical significance" (as is the unfortunate name), has anything to do with the "size/magnitude" of an effect? Because it doesn't. $\endgroup$ – Alecos Papadopoulos Jul 21 '14 at 19:11
1
$\begingroup$

If I understand correctly, your point, you are saying that, because the IV estimator gives only the "direct" effect of $x$ on $y$, and not the "indirect", of $x$ on $y$ through the errors (which you claim is non-linear), it does not give the correct interpretation of $\partial y /\partial x$. Instead, you suggest to use other methods such as kernel regression, lasso, etc... and say simply a model with endogeneity is wrong (so it is not clear what is your solution... use kernel/lasso for the model or discard the model?).

I see several issues with your argument. You notice correctly that in terms of prediction, OLS would be superior, so I assume your goal here is not to obtain the best prediction, but to estimate the effect of x on y precisely. If such is the case, the indirect effect through $\epsilon$ is generally of no interest for the model, since it is a sort of spurious effect, that precisely the IV estimator seeks to get rid of.

If you are interested in the interpretation of the IV estimate, there is an interesting discussion in the potential outcomes framework by Angrist and colleagues, who show that the IV identifies a "complier average treatment effect" (in the case dummy instrument and dummy endogeneous), so IV does correspond to a well defined quantity of direct interest.

Now regarding the other methods you suggest, note they no dot solve the problem of endogeneity. These are just different estimators for the same underlying model, and as such won't change the issue of endogeneity, and actually using these would be precisely sweeping under the rug the problem of endogeneity! So you would need to do kernel IV (a quite tough topic), or lasso IV, etc...

Coming finally to your last point that a model with endogeneity is a wrong model, here again you are getting confused between a model and its estimator... There are many models we cannot estimate (due to endogeneity, under-identification, too many parameters, etc), but saying that a model is wrong because our estimators are wrong is quite a controversial stand, and correspond to searching the keys under the light although you know they are somewhere else.

$\endgroup$
  • $\begingroup$ I agree that my post is confusing in regard of the "solution" (I would say: toss the model and find another one, either a linear model without endogeneity or a local method). However I disagree with your statement that the indirect effect through $\varepsilon$ is of no interest to the model: the fact that the model did not plan for this indirect effect does not make it spurious, but rather indicates that the model does not account for the full effect of $x$. And I must suppose the quantity of interest to which IV corresponds does not account for it either. $\endgroup$ – jubo Jul 21 '14 at 18:19
  • $\begingroup$ Regarding the assertion that a linear model with endogeneity is wrong, I still think it holds: $\endgroup$ – jubo Jul 21 '14 at 18:32
  • $\begingroup$ it is incapable of taking into account part of the effect of $x$ on $y$, because it supposes that $\varepsilon$ does not vary with $x$ (and IV comes down to taking the model seriously anyway). And I think the problem is not the estimator so much as the model: any linear model is interpreted as though $x$ did not affect $\varepsilon$, which is just wrong with endogeneity present. Any estimator taking the model seriously will rely on this violated hypothesis. $\endgroup$ – jubo Jul 21 '14 at 18:56
  • $\begingroup$ And my position is that local methods do not suffer from endogeneity, because they transform model (1) into $y^i = \beta_0^i + \beta_1^i x^i + \beta_2^i z^i + \varepsilon^i$, letting the parameters vary from one point to another, and thus do not impose any restriction on the form of $\frac{\partial y}{\partial x}$. So they solve endogeneity by relaxing the very hypothesis that made it a problem in the first place. $\endgroup$ – jubo Jul 21 '14 at 19:05
  • $\begingroup$ PS: my claim that the uncaptured effect is non-linear stems from the fact that I suppose any linear effect of $x$ will be captured by OLS or IV, but I am not fully sure that is true. $\endgroup$ – jubo Jul 21 '14 at 19:12
0
$\begingroup$

Here are Cameron and Trivedi (Microeconometrics, 2005) explaining the tight connection between IV consistency and correct model specification:

  • "The essential condition for consistency of IV is condition 1 in Section 4.8.6, that the instrument should be uncorrelated with the error term. No test is possible in the just-identified case. In the overidentified case a test of the overidentifying assumptions is possible (see Section 6.4.3). Rejection then could be due to either instrument endogeneity or model failure. Thus condition 1 is difficult to test directly and determining whether an instrument is exogenous is usually a subjective decision, albeit one often guided by economic theory."

  • "There is no formal test of instrument exogeneity that does not additionally test whether the regression equation is correctly specified. Instrument exogeneity inevitably relies on a priori information, such as that from economic or statistical theory."

So interpreting IV estimates as the true effects demands simple faith in a proposition that is absolutely untestable, because $\varepsilon$ is by definition unobserved, and on top of that any valid instrument will be correlated with $\hat \varepsilon_{OLS}$... does that sound scientific to anyone?

Unless there is solid external evidence that the model is correctly specified, and that $\frac{\partial y }{\partial x}$ has the form specified in the model, the error term counting for nothing, I would say it doesn't (and I tend to doubt that such evidence can be robustly found in the social sciences).

$\endgroup$
  • $\begingroup$ And a paper by Hall, Tavlas and Swamy (2011) The Non-existence of Instrumental Variables. $\endgroup$ – jubo Jul 23 '14 at 1:45
  • $\begingroup$ This paper basically says "let's express excluded variables as a function of included variables... why? because mathematically and in abstract, we can express anything as a function of anything else". Their crucial equation is $[3]$ for which they only provide the following argument: "assuming that the included variables are uncorrelated with the excluded variables would be meaningless" Why it would be meaningless? I wonder. In other words they assume without any convincing argument (mathematical, philosophical or whatever), what they want to prove. Sorry, but this is very bad science. $\endgroup$ – Alecos Papadopoulos Jul 23 '14 at 16:31
  • $\begingroup$ Equation (3) you are talking about is a generalized form of any regression model, taking into account the fact that some unobserved (excluded) variables have an influence on $y$. Excluded variables that are uncorrelated with the included ones would be caught up in the intercept and error term of a correctly-specified OLS model, so it is not a very interesting case. Not assuming that they are uncorrelated is another way of saying we stay in the general case (which allows uncorrelation as a special case). Φίλε, αυτό είναι πολύ καλή επιστήμη! $\endgroup$ – jubo Jul 25 '14 at 3:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.