Let's say that there exists some "true" relationship between $y$ and $x$ such that $y = ax + b + \epsilon$, where $a$ and $b$ are constants and $\epsilon$ is i.i.d normal noise. When I randomly generate data from that R code: x <- 1:100; y <- ax + b + rnorm(length(x)) and then fit a model like y ~ x, I obviously get reasonably good estimates for $a$ and $b$.

If I switch the role of the variables as in (x ~ y), however, and then rewrite the result for $y$ to be a function of $x$, the resulting slope is always steeper (either more negative or more positive) than that estimated by the y ~ x regression. I'm trying to understand exactly why that is and would appreciate it if anyone could give me an intuition as to what's going on there.

  • That's not true in general. Perhaps you're just seeing that in your data. Paste this code: y = rnorm(10); x = rnorm(10); lm(y~x); lm(x~y); into R several times and you'll find it goes both ways. – Macro Jan 3 '12 at 19:29
  • That's a bit different from what I was describing. In your example y wasn't a function of x at all, so there's not really any "slope" (the 'a' in my example). – Greg Aponte Jan 3 '12 at 19:35
  • lm(y~x) fits the model $y = \beta_{0} + \beta_{1}x + \varepsilon$ by least squares (equivalent to ML estimation when the errors are iid normal). There is a slope. – Macro Jan 3 '12 at 19:40
  • 1
    Your question is asked and answered (sort of) at stats.stackexchange.com/questions/13126 and stats.stackexchange.com/questions/18434. However, I believe nobody has yet contributed a simple, clear explanation of the relationships between (a) regression of $Y$ vs $X$, (b) regression of $X$ vs $Y$, (c) analysis of the correlation of $X$ and $Y$, (d) errors-in-variables regression of $X$ and $Y$, and (e) fitting a bivariate Normal distribution to $(X,Y)$. This would be a good place for such an exposition :-). – whuber Jan 3 '12 at 19:41
  • 2
    Of course Macro is correct: because x and y play equivalent roles in the question, which slope is more extreme is a matter of chance. However, geometry suggests (incorrectly) that when we reverse x and y in the regression, we should get the recipocal of the original slope. That never happens except when x and y are linearly dependent. This question can be interpreted as asking why. – whuber Jan 3 '12 at 19:50
up vote 17 down vote accepted

Given $n$ data points $(x_i,y_i), i = 1,2,\ldots n$, in the plane, let us draw a straight line $y = ax+b$. If we predict $ax_i+b$ as the value $\hat{y}_i$ of $y_i$, then the error is $(y_i-\hat{y}_i) = (y_i-ax_i-b)$, the squared error is $(y_i-ax_i-b)^2$, and the total squared error $\sum_{i=1}^n (y_i-ax_i-b)^2$. We ask

What choice of $a$ and $b$ minimizes $S =\displaystyle\sum_{i=1}^n (y_i-ax_i-b)^2$?

Since $(y_i-ax_i-b)$ is the vertical distance of $(x_i,y_i)$ from the straight line, we are asking for the line such that the sum of the squares of the vertical distances of the points from the line is as small as possible. Now $S$ is a quadratic function of both $a$ and $b$ and attains its minimum value when $a$ and $b$ are such that $$\begin{align*} \frac{\partial S}{\partial a} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-x_i) &= 0\\ \frac{\partial S}{\partial b} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-1) &= 0 \end{align*}$$ From the second equation, we get $$b = \frac{1}{n}\sum_{i=1}^n (y_i - ax_i) = \mu_y - a\mu_x$$ where $\displaystyle \mu_y = \frac{1}{n}\sum_{i=1}^n y_i, ~ \mu_x = \frac{1}{n}\sum_{i=1}^n x_i$ are the arithmetic average values of the $y_i$'s and the $x_i$'s respectively. Substituting into the first equation, we get $$ a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}. $$ Thus, the line that minimizes $S$ can be expressed as $$y = ax+b = \mu_y + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}\right) (x - \mu_x), $$ and the minimum value of $S$ is $$S_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}.$$

If we interchange the roles of $x$ and $y$, draw a line $x = \hat{a}y + \hat{b}$, and ask for the values of $\hat{a}$ and $\hat{b}$ that minimize $$T = \sum_{i=1}^n (x_i - \hat{a}y_i - \hat{b})^2,$$ that is, we want the line such that the sum of the squares of the horizontal distances of the points from the line is as small as possible, then we get

$$x = \hat{a}y+\hat{b} = \mu_x + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}\right) (y - \mu_y) $$ and the minimum value of $T$ is $$T_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}.$$

Note that both lines pass through the point $(\mu_x,\mu_y)$ but the slopes are $$a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2},~~ \hat{a}^{-1} = \frac{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}$$ are different in general. Indeed, as @whuber points out in a comment, the slopes are the same when all the points $(x_i,y_i)$ lie on the same straight line. To see this, note that $$\hat{a}^{-1} - a = \frac{S_{\min}}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y} = 0 \Rightarrow S_{\min} = 0 \Rightarrow y_i=ax_i+b, i=1,2,\ldots, n. $$

  • Thanks! abs(correlation) < 1 accounts for why the slope was systematically steeper in the reversed case. – Greg Aponte Jan 3 '12 at 20:45
  • (+1) but I added an answer with just an illustration of what you just said, as I have a geometric mind :) – Elvis Jan 3 '12 at 21:04
  • Class reply (+1) – Digio Oct 19 '15 at 10:26

Just to illustrate Dilip’s answer: on the following pictures,

  • the black dots are data points ;
  • on the left, the black line is the regression line obtained by y ~ x, which minimize the squares of the length of the red segments;
  • on the right, the black line is the regression line obtained by x ~ y, which minimize the squares of the length of the red segments.

regression lines

Edit (least rectangles regression)

If there is no natural way to chose a "response" and a "covariate", but rather the two variables are interdependent you may wish to conserve a symmetrical role for $y$ and $x$; in this case you can use "least rectangles regression."

  • write $Y = aX + b + \epsilon$, as usual;
  • denote $\hat y_i = a x_i + b$ and $\hat x_i = {1\over a} (y_i - b)$ the estimations of $Y_i$ conditional to $X = x_i$ and of $X_i$ conditional to $Y = y_i$;
  • minimize $\sum_i | x_i - \hat x_i | \cdot | y_i - \hat y_i|$, which leads to $$\hat y = \mathrm{sign}\left(\mathrm{cov}(x,y)\right){\hat\sigma_y \over \hat\sigma_x} (x-\overline x) + \overline y. $$

Here is an illustration with the same data points, for each point, a "rectangle" is computed as the product of the length of two red segments, and the sum of rectangles is minimized. I don’t know much about the properties of this regression and I don’t find much with google.

least rectangles

  • 10
    Some notes: (1) Unless I am mistaken, it seems that the "least rectangles regression" is equivalent to the solution obtained from taking the first principal component on the matrix $\mathbf X = (\mathbf y, \mathbf x)$ after centering and rescaling to have unit variance and then backsubstituting. (cont.) – cardinal Jan 3 '12 at 23:39
  • 10
    (cont.) (2) Viewed this way, it is easy to see that this "least rectangles regression" is equivalent to a form of orthogonal (or total) least squares and, thus, (3) A special case of Deming regression on the centered, rescaled vectors taking $\delta = 1$. Orthogonal least squares can be considered as "least-circles regression". – cardinal Jan 3 '12 at 23:41
  • 2
    @cardinal Very interesting comments! (+1) I believe major axis (minimizing perpendicular distances between reg. line and all the points, à la PCA) or reduced major axis regression, or type II regression as exemplified in the lmodel2 R package by P Legendre, are also relevant here since those techniques are used when it's hard to tell what role (response or predictor) plays each variable or when we want to account for measurement errors. – chl Jan 4 '12 at 10:32
  • 1
    @chl: (+1) Yes, I believe you are right and the Wikipedia page on total least squares lists several other names for the same procedure, not all of which I am familiar with. It appears to go back to at least R. Frisch, Statistical confluence analysis by means of complete regression systems, Universitetets Økonomiske Instituut, 1934 where it was called diagonal regression. – cardinal Jan 4 '12 at 13:43
  • 2
    @cardinal I should have been more careful when reading the Wikipedia entry... For future reference, here is a picture taken from Biostatistical Design and Analysis Using R, by M. Logan (Wiley, 2010; Fig. 8.4, p. 174), which summarizes the different approaches, much like Elvis's nice illustrations. – chl Jan 8 '12 at 12:22

Just a brief note on why you see the slope smaller for one regression. Both slopes depend on three numbers: standard deviations of $x$ and $y$ ($s_{x}$ and $s_{y}$), and correlation between $x$ and $y$ ($r$). The regression with $y$ as response has slope $r\frac{s_{y}}{s_{x}}$ and the regression with $x$ as response has slope $r\frac{s_{x}}{s_{y}}$, hence the ratio of the first slope to the reciprocal of the second is equal to $r^2\leq 1$.

So the greater the proportion of variance explained, the closer the slopes obtained from each case. Note that the proportion of variance explained is symmetric and equal to the squared correlation in simple linear regression.

It becomes interesting when there is also noise on your inputs (which we could argue is always the case, no command or observation is ever perfect).

I have built some simulations to observe the phenomenon, based on a simple linear relationship $x = y$, with Gaussian noise on both x and y. I generated the observations as follows (python code):

x = np.linspace(0, 1, n)
y = x

x_o = x + np.random.normal(0, 0.2, n)
y_o = y + np.random.normal(0, 0.2, n)

See the different results (odr here is orthogonal distance regression, i.e. the same as least rectangles regression):

enter image description here

All the code is in there:

https://gist.github.com/jclevesque/5273ad9077d9ea93994f6d96c20b0ddd

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.