# Regression to the mean

I recently asked this question

In regression model with random regressors

$$(1) \ \ y = a + bx + e$$

can I change the equation to

$$(2) \ \ x = (-a/b) + (1/b)y + (-1/b)e$$

and consistently estimate $$(1/b)$$ with OLS?

which is already answered here whats the difference between regression of x on y versus y on x and related here. Very interesting post but I would like to ask the following follow up question:

I guess we can all agree that (1) and (2) are mathematically equivalent. I then noticed that if I simulate equation (1) by

1. Drawing x and drawing e with x independently of x and then calulate y using formula (1) the OLS estimator regressing y on x consistently estimates a and b.

However if simulate by

1. Drawing y and drawing e indenpently and the calculate x according to (2) then the OLS estimator regressing x on y consistently estimates $$\lambda_0=(-a/b)$$ and $$\lambda_1 := 1/b$$.

The question then is does this not mean that the model statement (1) is somehow incomplete in the sense that a more complete model statement would be

$$y = a + b x + e \wedge x \perp e$$ versus

$$y = a + b x + e \wedge y \perp e$$

because for the latter model it would exactly be necessary to use OLS as associated with the equation

$$(2) \ \ x = (-a/b) + (1/b)y + (-1/b)e$$

to get consistent estimates. So my question boils down to when we simply write $$y = a + bx + e$$ the statement is somewhat incomplete missing the variable dependencies?

• Could you explain what you mean by "$e \wedge x \perp e$"? The operators have ambiguous interpretations and even the order of operations is ambiguous. Please note that in OLS, $x$ can be taken as arbitrary: it need not be considered a random variable. As far as "variable dependencies" goes, you are correct that you haven't fully specified the OLS assumptions. They usually include that the $e_i$ are iid with zero mean and finite variance. See stats.stackexchange.com/questions/16381. – whuber Jan 9 '20 at 21:24
• It was just my way of formally saying I use the model $y = a +bx + e$ and $x \perp e$ to denote $x$ independent of $e$ so the $\wedge$ was just logical "and". – kenxavierfractal Jan 9 '20 at 21:34
• Also I am not sure I know what "$x$ can be taken as arbitrary" actually means other than they are some arbitrary constants if in fixed regressor setup. Still the point I guess would then be that even in a fixed regressor setup $y$ have different status that $x$ depending on what side of the identity sig they appear on. If I wrote the $y$'s on the RHS one would standardly read "$y$'s are arbitrary constants" right? – kenxavierfractal Jan 9 '20 at 21:40
• As you saw in comments to your previous question, which side a variable appears on is irrelevant: there is an inherent asymmetry between $x$ and $y$ evident in their conditional distributions. As far as I can tell, the thread I referenced above answers all your questions here. – whuber Jan 9 '20 at 22:03
• Ok, so perhaps in the random regressor set the correct thing to say is simply the random vector $(y,x,e)$ satisfies the relationship $y = a + bx +e$. If $cov(x,e)=0$ then it follows that $b=var(x)^{-1}cov(x,y)$ which OLS $\hat b := S_{xx}^{-1}S_{xy}$ consistently estimates. And $\hat \lambda_1 := S_{yy}^{-1}S_{yx}$ converge in probability to $Var(y)^{-1}cov(yx) = (1/b) -(1/b) var(e)$ hence biased estimate of $(1/b)$. – kenxavierfractal Jan 10 '20 at 17:00

The issue here is that equation $$(1)$$ does not fully specify the regression model, so yes, that equation is an incomplete statement of the model. In order to fully specify the regression model, you need to specify the distribution of $$e$$ conditional on $$x$$. For a homoscedastic Gaussian linear regression model, the defining equations are:
$$y = a + b x + e \quad \quad \quad \quad \quad e | x \sim \text{N}(0, \sigma^2).$$
The second equation does indeed imply that $$e \perp x$$, but it is not generally true that $$e \perp y$$. Thus, while you can certainly manipulate equation $$(1)$$ into $$(2)$$ (so long as $$b \neq 0$$), this is not sufficient to give you a linear regression model for $$x$$ in terms of $$y$$. This is what whuber is referring to in the comments when he says that the regression model is inherently asymmetric.
It is possible to obtain a symmetric case by making the much stronger assumption that $$(x,y)$$ are jointly normally distributed. In this case it is possible to write either variable in terms of the other using the form of a linear regression model. This assumption is stronger than the assumptions of a linear regression model on one variable, which only specifies the conditional distribution of that variable given the other variable.
• If $(x,y)$ is MVN you say "it is possible to write either variable in terms of the other using the form of a linear regression model", but is this not possible for any pair $(x,y)$ of variables simply using linear projections? For example $y =x^\top \alpha + u$ with $x$ including 1 and then $\alpha := \mathbb E[xx^\top]^{-1}\mathbb E[xy]$ and $u:= y - x^\top \alpha$. The $cov(x,u)=0$ but perhaps using a MVN you have the stronger property that $\mathbb E[y \lvert x] = x^\top \alpha$ – kenxavierfractal Jan 10 '20 at 18:21