Let $X$ and $Y$ be mean 0 and variance 1 random variables such that we choose $\alpha$ and $\beta$ to minimise

$$\mathbb{E}(X-\beta Y)^2$$


$$\mathbb{E}(Y-\alpha X)^2$$

after not so difficult derivation, I arrive at $\alpha = \mathbb{E}(XY)/\mathbb{E}X^2$ and $\beta = \mathbb{E}(XY)/\mathbb{E}Y^2$, so $\alpha = \beta$.

This seems very strange, because if $y=mx$ is regression line, then surely $x = \frac{1}{m }y$.

  • $\begingroup$ (X - Y)^2 = (Y - X)^2 for each X and Y. Basically it's the same function (expand the expressions for prove). That's why you get alpha = beta $\endgroup$ – Ivan Oct 23 '15 at 12:12
  • $\begingroup$ @Ivan sorry, you are going too fast for me. How are they the same function? The coefficient is in front of $X$ in one of the expression and $Y$ in the other. $\endgroup$ – Lost1 Oct 23 '15 at 12:15
  • $\begingroup$ that's minimization problem. In best case you have E(X - \betaY)^2 = 0 expanding the expression X^2 - 2X\betaY + Y^2 = 0 \beta = (X^2 - 2xy + y^2) and E(Y-\alphaX)^2 = 0 \alpha = (Y^2 - 2xy + X^2) witch is basically the same $\endgroup$ – Ivan Oct 23 '15 at 12:19
  • $\begingroup$ @Ivan when you expand you should get $X^2+2\beta XY +\beta^2 Y^2$? why has the beta's disappeared? I do not follow this. $\endgroup$ – Lost1 Oct 23 '15 at 12:21
  • 1
    $\begingroup$ What do you precisely mean by "because if $y=mx$ is regression line, then surely $x = \frac{1}{m }y$". Is it to say that, if $\hat\beta$ is the regression coefficient of a regression of $y$ on $x$, $\hat\beta^{-1}$ is the coefficient of the regression of $x$ on $y$? $\endgroup$ – Christoph Hanck Oct 23 '15 at 14:07

So I will answer my own question. The reason the two regression lines are different is this:

The regression line of $Y$ against $X$ minimises the total squared distance of the $Y$ COORDINATES to the $y$ COORDINATES of the line, where as $X$ against $Y$ minimise the total squared distance of $X$ COORDINATES to the $x$ COORDINATES line.

The confusion I had with regards to the line should be unique comes from the mistaken assumption the regression line minimise the total squared (PERPENDICULAR) distance of each point to the regression line. If this had been the case, my assumption would have been true. However, the two measure of distance never coincide unless there is perfect correlation.

| cite | improve this answer | |
  • $\begingroup$ I think @Christoph Hanck's link provides the answer. $\endgroup$ – mr.rox Oct 23 '15 at 15:47
  • $\begingroup$ @mr.rox well, ive realised before i saw the link. it doesn't matter any more. $\endgroup$ – Lost1 Oct 23 '15 at 15:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.