2
$\begingroup$

I recently asked this question

In regression model with random regressors

$$(1) \ \ y = a + bx + e$$

can I change the equation to

$$(2) \ \ x = (-a/b) + (1/b)y + (-1/b)e$$

and consistently estimate $(1/b)$ with OLS?

which is already answered here whats the difference between regression of x on y versus y on x and related here. Very interesting post but I would like to ask the following follow up question:

I guess we can all agree that (1) and (2) are mathematically equivalent. I then noticed that if I simulate equation (1) by

  1. Drawing x and drawing e with x independently of x and then calulate y using formula (1) the OLS estimator regressing y on x consistently estimates a and b.

However if simulate by

  1. Drawing y and drawing e indenpently and the calculate x according to (2) then the OLS estimator regressing x on y consistently estimates $\lambda_0=(-a/b)$ and $\lambda_1 := 1/b$.

The question then is does this not mean that the model statement (1) is somehow incomplete in the sense that a more complete model statement would be

$$y = a + b x + e \wedge x \perp e$$ versus

$$y = a + b x + e \wedge y \perp e$$

because for the latter model it would exactly be necessary to use OLS as associated with the equation

$$(2) \ \ x = (-a/b) + (1/b)y + (-1/b)e$$

to get consistent estimates. So my question boils down to when we simply write $$y = a + bx + e$$ the statement is somewhat incomplete missing the variable dependencies?

Improve this question
$\endgroup$
7
  • $\begingroup$ Could you explain what you mean by "$e \wedge x \perp e$"? The operators have ambiguous interpretations and even the order of operations is ambiguous. Please note that in OLS, $x$ can be taken as arbitrary: it need not be considered a random variable. As far as "variable dependencies" goes, you are correct that you haven't fully specified the OLS assumptions. They usually include that the $e_i$ are iid with zero mean and finite variance. See stats.stackexchange.com/questions/16381. $\endgroup$ – whuber♦ Jan 9 '20 at 21:24
  • $\begingroup$ It was just my way of formally saying I use the model $y = a +bx + e$ and $x \perp e$ to denote $x$ independent of $e$ so the $\wedge$ was just logical "and". $\endgroup$ – kenxavierfractal Jan 9 '20 at 21:34
  • $\begingroup$ Also I am not sure I know what "$x$ can be taken as arbitrary" actually means other than they are some arbitrary constants if in fixed regressor setup. Still the point I guess would then be that even in a fixed regressor setup $y$ have different status that $x$ depending on what side of the identity sig they appear on. If I wrote the $y$'s on the RHS one would standardly read "$y$'s are arbitrary constants" right? $\endgroup$ – kenxavierfractal Jan 9 '20 at 21:40
  • 2
    $\begingroup$ As you saw in comments to your previous question, which side a variable appears on is irrelevant: there is an inherent asymmetry between $x$ and $y$ evident in their conditional distributions. As far as I can tell, the thread I referenced above answers all your questions here. $\endgroup$ – whuber♦ Jan 9 '20 at 22:03
  • $\begingroup$ Ok, so perhaps in the random regressor set the correct thing to say is simply the random vector $(y,x,e)$ satisfies the relationship $y = a + bx +e$. If $cov(x,e)=0$ then it follows that $b=var(x)^{-1}cov(x,y)$ which OLS $\hat b := S_{xx}^{-1}S_{xy}$ consistently estimates. And $\hat \lambda_1 := S_{yy}^{-1}S_{yx}$ converge in probability to $Var(y)^{-1}cov(yx) = (1/b) -(1/b) var(e)$ hence biased estimate of $(1/b)$. $\endgroup$ – kenxavierfractal Jan 10 '20 at 17:00
3
$\begingroup$

The issue here is that equation $(1)$ does not fully specify the regression model, so yes, that equation is an incomplete statement of the model. In order to fully specify the regression model, you need to specify the distribution of $e$ conditional on $x$. For a homoscedastic Gaussian linear regression model, the defining equations are:

$$y = a + b x + e \quad \quad \quad \quad \quad e | x \sim \text{N}(0, \sigma^2).$$

The second equation does indeed imply that $e \perp x$, but it is not generally true that $e \perp y$. Thus, while you can certainly manipulate equation $(1)$ into $(2)$ (so long as $b \neq 0$), this is not sufficient to give you a linear regression model for $x$ in terms of $y$. This is what whuber is referring to in the comments when he says that the regression model is inherently asymmetric.

It is possible to obtain a symmetric case by making the much stronger assumption that $(x,y)$ are jointly normally distributed. In this case it is possible to write either variable in terms of the other using the form of a linear regression model. This assumption is stronger than the assumptions of a linear regression model on one variable, which only specifies the conditional distribution of that variable given the other variable.

Improve this answer
$\endgroup$
3
  • $\begingroup$ If $(x,y)$ is MVN you say "it is possible to write either variable in terms of the other using the form of a linear regression model", but is this not possible for any pair $(x,y)$ of variables simply using linear projections? For example $y =x^\top \alpha + u$ with $x$ including 1 and then $\alpha := \mathbb E[xx^\top]^{-1}\mathbb E[xy]$ and $u:= y - x^\top \alpha$. The $cov(x,u)=0$ but perhaps using a MVN you have the stronger property that $\mathbb E[y \lvert x] = x^\top \alpha$ $\endgroup$ – kenxavierfractal Jan 10 '20 at 18:21
  • $\begingroup$ which can probably be shown using this page MVN-conditional distribution? $\endgroup$ – kenxavierfractal Jan 10 '20 at 18:21
  • $\begingroup$ @kenxavierfractal: Again, you seem to be missing the requirement that the linear regression model is more than just the linear equation. You can always invert the linear projection, but the point is that this is not enough to get a linear regression model the other way, unless you had a strong enough assumption on the distribution the obtain the corresponding distributional requirements on the error term. $\endgroup$ – Ben Jan 11 '20 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.