# Endogeneity and Regression Interpretation: how does it work? is it possible?

My question is not particularly straightforward. I will explain my reasoning, and then give the question at the end to avoid any confusion.

$$\ Y_{i} = \beta_{0}+\beta_{1}X_{i}+\varepsilon_{i}$$

Suppose that this our population regression function. We note that $$\ \beta_{0}$$ and $$\ \beta_{1}$$ are not arbitrary. This means that the error term $$\ \varepsilon_{i}$$ has some structure to it. In particular, it reflects the deviation of $$Y_{i}$$ from its expected value. To put it even more clearly, we have that $$E[Y_{i}|X_{i}]= \beta_{0}+\beta_{1}X_{i}$$. What I am trying to say here is that our population parameters must take some non-arbitrary value, otherwise we might as well define our errors to make the equation balance and pick them randomly.

Now suppose we want to take a sample and estimate the above population regression function. But before doing so, we want to work out what the true value of $$\beta_{0}$$ and $$\beta_{1}$$ are. To do this we appeal to the mean square error as our guide (I drop the subscripts for simplicity of notation):

$$MSE(b_{0}, b_{1}) = E[(Y - (b_{0}+b_{1}X))^2] = E[Y^2] -2b_{0}E[Y]-2b_{1}E[XY]+E[(b_{0}+b_{1}X)^2]$$

$$MSE(b_{0}, b_{1}) = E[Y^2] -2b_{0}E[Y]-2b_{1}Cov(X,Y) - 2b_{1}E[X]E[Y]+b_{0}^2+2b_{0}b_{1}E[X]+b_{1}^2Var[X]+b_{1}^2(E[X])^2$$

I have skipped some of the algebra because it is not really important at this point. Now I take first order conditions with respect to my two variables:

$$\frac{\partial MSE(b_{0},b_{1})}{\partial b_0}= -2E[Y]+2b_{0}+2b_{1}E[X]$$

$$\frac{\partial MSE(b_{0},b_{1})}{\partial b_1}= -2Cov(X,Y)-2E[X]E[Y]+2b_{0}E[X]+2b_{1}Var[X]+2b_{1}(E[X])^2$$

Setting the partial derivates equal to zero and solving for each parameter gives:

$$\beta_{0}=E[Y]-\beta_{1}E[X]$$

$$\beta_{1}= \frac{Cov[X,Y]}{Var[X]}$$

Now that we have the equations of our true population parameters, we can consider the issue at hand, otherwise known as my question! Let's try to check the correlation between our explanatory variables and the error term defined above:

$$\ E[X\varepsilon] = E[X(Y-(\beta_{0}+\beta_{1}X)] = E[XY - X\beta_{0}-\beta_{1}X^2]$$

Using some simple algebraic manipulations we get that:

$$E[X\varepsilon]= E[XY]- E[X\beta_{0}+\beta_{1}X^2]$$

$$E[X\varepsilon]= E[XY]- E[X(E[Y]-\beta_{1}E[X])+\beta_{1}X^2]$$

$$E[X\varepsilon]= E[XY]- E[XE[Y]-X\beta_{1}E[X]+\beta_{1}X^2]$$

$$E[X\varepsilon]= E[XY]- E[Y]E[X] + E[X\beta_{1}E[X]] - E[\beta_{1}X^2]$$

$$E[X\varepsilon]= E[XY]- E[Y]E[X] + \beta_{1}E[X]^2 - E[\beta_{1}X^2]$$

$$E[X\varepsilon]= E[XY]- E[Y]E[X] + \beta_{1}E[X]^2 - \beta_{1}E[X^2]$$

$$E[X\varepsilon]= E[XY]- E[Y]E[X] -\beta_{1}(E[X^2] - E[X]^2)$$

$$E[X\varepsilon]= Cov[X,Y]-\beta_{1}Var[X]$$

$$E[X\varepsilon]= Cov[X,Y]-\frac{Cov[X,Y]}{Var[X]}Var[X]$$

$$E[X\varepsilon]= 0$$

Now what we have proved is the by construction the parameters of the population regression function are defined in such a way that the covariance between the explanatory variables and errors is equal to zero.

Now bearing all of the above in mind, herein lies my question: how is endogeneity mathematically possible? Is it that my interpretation of the regression is wrong? I have heard others mention that there is some sort of causal interpretation of regression that must be taken, though I am not really sure what that means, or what that is. Any insights?

P.S. Thank you to those that have answered my other questions about regression analysis previously, I am learning a lot!

You are correct. In the linear regression $$\mathbb{E}[X\varepsilon] = 0$$ by construction. I talked about this in another post here

https://stats.stackexchange.com/a/550305/261146

• You need an "externally defined" quantity of interest.
• Based on this external definition you decide whether OLS satisfies exogeneity.

1. An illustrative example

I think the best example is easily selection into treatment, the "causal case".

Suppose that an individual is assigned to a treatment $$D \in \{0,1\}$$. If the individual is part of the control she will have an outcome $$Y_0$$. If she is treated, her outcome is $$Y_1$$. The outcome observed by the researcher is: \begin{align*} Y &= DY_1 + (1-D)Y_0 \\ &= Y_0 + D(Y_1 - Y_0) \end{align*}The variables $$(Y_1,Y_0)$$ are known as "potential outcomes". An individual cannot belong to both groups at the same time, which means that one of the outcomes is never observed. The best the researcher can do is find two groups with similar characteristics and different treatment status to compare their outcomes. The average treatment effect is defined as $$\tau = \mathbb{E}[Y_1 - Y_0]$$ This quantity is important in medical trials and many situations because it indicates the population-level treatment effect.

Can we identify $$\tau$$ from an OLS regression of $$Y$$ on $$D$$?

2. Running an OLS estimator:

The OLS regression takes the form: $$Y = \mu_{OLS} + \tau_{OLS} D + \varepsilon$$

The coefficient $$(\mu_{OLS},\tau_{OLS})$$ are constructed to minimize MSE, and will satisfy the exogeneity condition.

The OLS coefficient of $$\tau$$ is equal to \begin{align*} \tau_{OLS} &= \frac{\mathbb{E}[DY]}{\mathbb{E}[D]} - \frac{\mathbb{E}[(1-D)Y]}{\mathbb{E}[1-D]} \end{align*}

3. Does OLS work, i.e. $$\tau_{OLS} = \tau$$?

• By substituting the definition of $$Y$$ with potential outcomes, this is equal to $$\tau_{OLS} = \frac{\mathbb{E}[DY_1]}{\mathbb{E}[D]} - \frac{\mathbb{E}[(1-D)Y_0]}{\mathbb{E}[1-D]}$$ Case 1: (Random assignment) When $$D$$ is randomly assigned, $$D$$ is independent of $$(Y_1,Y_0)$$. This means that $$\mathbb{E}[DY_1] = \mathbb{E}[D]\mathbb{E}[Y_1]$$ and $$\mathbb{E}[DY_0] = \mathbb{E}[D]\mathbb{E}[Y_0]$$. We can then readily verify that $$\tau_{OLS} = \mathbb{E}[Y_1] - \mathbb{E}[Y_0] = \tau$$.

Case 2: (Self-selection) Suppose that $$D = 1$$ if and only if $$Y_1 > Y_0$$. This is situation where individuals only participate if they (correctly) assess that they will receive some benefit. In this case the terms may not cancel out and $$\tau_{OLS} \ne \tau$$.

3. Insights

• We can always define the potential outcomes as linear model:

$$Y = \mu + \tau D + U$$ where $$U = (Y_0 - \mu) + D(Y_1 - Y_0 - \tau)$$ and $$\mu = \mathbb{E}[Y_0]$$.

• The endogeneity condition in this case requires that

\begin{align*} \mathbb{E}[ D U ] &= \mathbb{E}[D(Y_1 - Y_0 - \tau)] = 0 \\ \mathbb{E}[ U ] &= \mathbb{E}[Y_0 - \mu + D(Y_1 - Y_0 - \tau)] = 0 \\ \end{align*}

• This may fail if $$D$$ is correlated with $$(Y_1,Y_0)$$.
• The OLS minimizes the MSE criterion. It is not guaranteed to produce anything else.
• We sometimes call the parameters $$(\mu_{OLS},\tau_{OLS})$$ "pseudo-parameters" because they solve the equation $$\mathbb{E}[X(Y - \mu_{OLS} - D \tau_{OLS})] = 0$$, where $$X = (1,D)$$.
• However, depending on the "external" model, these may or may not have a "useful" interpretation.
• To make sure I have understood. We distinguish the "true model" (whatever that is) and the OLS population regression function, which we want to estimate using sample data. In other words, there are essentially three models: (1) 'True Model' which encodes causation (2) Hypothesized Regression Function (could be OLS) and (3) Sample Regression Function. The endogeneity problem arises because, due to some external issue (e.g. self-selection above), the 'true' model does not match up with our 'Hypothesized' model. I have been excessively brief, but hopefully in the right direction. Commented Nov 3, 2021 at 21:57
• To put it, perhaps, another way, the population OLS parameters are one way to represent the 'true' causal model, and so we have to make assumptions (external definitions etc.) to make sure that it 'fits'. But this also raises a further question. When trying to show that the sample OLS estimates are unbiased, we get the following expression: $\beta + (X'X)^-1 X'\varepsilon$ This will clearly not refer to our "hypothesized PRF with OLS parameters but the "true" model, right? Commented Nov 3, 2021 at 22:13
• I think you have the right idea. The assumptions are the ones that make sure that the Hypothesized Regression Function (HRF) aligns with the "True model". The sample OLS estimates are unbiased estimators of the HRF coefficients. They may not necessarily recover the parameters of the true model unless the assumptions are satisfied. Commented Nov 3, 2021 at 22:26
• I would also add that in this example, whether $D$ is correlated with $(Y_1,Y_0)$ is not something the analyst can control. Therefore, the same OLS model may or may not recover the "true" model, depending on how the data is produced. This is why running RCTs (randomized controlled trials) is so important in medicine, as opposed to just using convenience sampling, to ensure this holds by design. Commented Nov 3, 2021 at 22:28
• That’s another good example. Commented Nov 4, 2021 at 14:52