# Effect of switching response and explanatory variable in simple linear regression

Let's say that there exists some "true" relationship between $y$ and $x$ such that $y = ax + b + \epsilon$, where $a$ and $b$ are constants and $\epsilon$ is i.i.d normal noise. When I randomly generate data from that R code: x <- 1:100; y <- ax + b + rnorm(length(x)) and then fit a model like y ~ x, I obviously get reasonably good estimates for $a$ and $b$.

If I switch the role of the variables as in (x ~ y), however, and then rewrite the result for $y$ to be a function of $x$, the resulting slope is always steeper (either more negative or more positive) than that estimated by the y ~ x regression. I'm trying to understand exactly why that is and would appreciate it if anyone could give me an intuition as to what's going on there.

• That's not true in general. Perhaps you're just seeing that in your data. Paste this code: y = rnorm(10); x = rnorm(10); lm(y~x); lm(x~y); into R several times and you'll find it goes both ways. – Macro Jan 3 '12 at 19:29
• That's a bit different from what I was describing. In your example y wasn't a function of x at all, so there's not really any "slope" (the 'a' in my example). – Greg Aponte Jan 3 '12 at 19:35
• lm(y~x) fits the model $y = \beta_{0} + \beta_{1}x + \varepsilon$ by least squares (equivalent to ML estimation when the errors are iid normal). There is a slope. – Macro Jan 3 '12 at 19:40
• Your question is asked and answered (sort of) at stats.stackexchange.com/questions/13126 and stats.stackexchange.com/questions/18434. However, I believe nobody has yet contributed a simple, clear explanation of the relationships between (a) regression of $Y$ vs $X$, (b) regression of $X$ vs $Y$, (c) analysis of the correlation of $X$ and $Y$, (d) errors-in-variables regression of $X$ and $Y$, and (e) fitting a bivariate Normal distribution to $(X,Y)$. This would be a good place for such an exposition :-). – whuber Jan 3 '12 at 19:41
• Of course Macro is correct: because x and y play equivalent roles in the question, which slope is more extreme is a matter of chance. However, geometry suggests (incorrectly) that when we reverse x and y in the regression, we should get the recipocal of the original slope. That never happens except when x and y are linearly dependent. This question can be interpreted as asking why. – whuber Jan 3 '12 at 19:50

Given $n$ data points $(x_i,y_i), i = 1,2,\ldots n$, in the plane, let us draw a straight line $y = ax+b$. If we predict $ax_i+b$ as the value $\hat{y}_i$ of $y_i$, then the error is $(y_i-\hat{y}_i) = (y_i-ax_i-b)$, the squared error is $(y_i-ax_i-b)^2$, and the total squared error $\sum_{i=1}^n (y_i-ax_i-b)^2$. We ask

What choice of $a$ and $b$ minimizes $S =\displaystyle\sum_{i=1}^n (y_i-ax_i-b)^2$?

Since $(y_i-ax_i-b)$ is the vertical distance of $(x_i,y_i)$ from the straight line, we are asking for the line such that the sum of the squares of the vertical distances of the points from the line is as small as possible. Now $S$ is a quadratic function of both $a$ and $b$ and attains its minimum value when $a$ and $b$ are such that \begin{align*} \frac{\partial S}{\partial a} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-x_i) &= 0\\ \frac{\partial S}{\partial b} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-1) &= 0 \end{align*} From the second equation, we get $$b = \frac{1}{n}\sum_{i=1}^n (y_i - ax_i) = \mu_y - a\mu_x$$ where $\displaystyle \mu_y = \frac{1}{n}\sum_{i=1}^n y_i, ~ \mu_x = \frac{1}{n}\sum_{i=1}^n x_i$ are the arithmetic average values of the $y_i$'s and the $x_i$'s respectively. Substituting into the first equation, we get $$a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}.$$ Thus, the line that minimizes $S$ can be expressed as $$y = ax+b = \mu_y + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}\right) (x - \mu_x),$$ and the minimum value of $S$ is $$S_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2}.$$

If we interchange the roles of $x$ and $y$, draw a line $x = \hat{a}y + \hat{b}$, and ask for the values of $\hat{a}$ and $\hat{b}$ that minimize $$T = \sum_{i=1}^n (x_i - \hat{a}y_i - \hat{b})^2,$$ that is, we want the line such that the sum of the squares of the horizontal distances of the points from the line is as small as possible, then we get

$$x = \hat{a}y+\hat{b} = \mu_x + \left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}\right) (y - \mu_y)$$ and the minimum value of $T$ is $$T_{\min} = \frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2\right] \left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2\right] - \left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}.$$

Note that both lines pass through the point $(\mu_x,\mu_y)$ but the slopes are $$a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{ \left( \frac{1}{n}\sum_{i=1}^n x_i^2\right) -\mu_x^2},~~ \hat{a}^{-1} = \frac{ \left( \frac{1}{n}\sum_{i=1}^n y_i^2\right) -\mu_y^2}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}$$ are different in general. Indeed, as @whuber points out in a comment, the slopes are the same when all the points $(x_i,y_i)$ lie on the same straight line. To see this, note that $$\hat{a}^{-1} - a = \frac{S_{\min}}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y} = 0 \Rightarrow S_{\min} = 0 \Rightarrow y_i=ax_i+b, i=1,2,\ldots, n.$$

• Thanks! abs(correlation) < 1 accounts for why the slope was systematically steeper in the reversed case. – Greg Aponte Jan 3 '12 at 20:45
• (+1) but I added an answer with just an illustration of what you just said, as I have a geometric mind :) – Elvis Jan 3 '12 at 21:04
• Class reply (+1) – Digio Oct 19 '15 at 10:26

Just to illustrate Dilip’s answer: on the following pictures,

• the black dots are data points ;
• on the left, the black line is the regression line obtained by y ~ x, which minimize the squares of the length of the red segments;
• on the right, the black line is the regression line obtained by x ~ y, which minimize the squares of the length of the red segments. Edit (least rectangles regression)

If there is no natural way to chose a "response" and a "covariate", but rather the two variables are interdependent you may wish to conserve a symmetrical role for $y$ and $x$; in this case you can use "least rectangles regression."

• write $Y = aX + b + \epsilon$, as usual;
• denote $\hat y_i = a x_i + b$ and $\hat x_i = {1\over a} (y_i - b)$ the estimations of $Y_i$ conditional to $X = x_i$ and of $X_i$ conditional to $Y = y_i$;
• minimize $\sum_i | x_i - \hat x_i | \cdot | y_i - \hat y_i|$, which leads to $$\hat y = \mathrm{sign}\left(\mathrm{cov}(x,y)\right){\hat\sigma_y \over \hat\sigma_x} (x-\overline x) + \overline y.$$

Here is an illustration with the same data points, for each point, a "rectangle" is computed as the product of the length of two red segments, and the sum of rectangles is minimized. I don’t know much about the properties of this regression and I don’t find much with google. • Some notes: (1) Unless I am mistaken, it seems that the "least rectangles regression" is equivalent to the solution obtained from taking the first principal component on the matrix $\mathbf X = (\mathbf y, \mathbf x)$ after centering and rescaling to have unit variance and then backsubstituting. (cont.) – cardinal Jan 3 '12 at 23:39
• (cont.) (2) Viewed this way, it is easy to see that this "least rectangles regression" is equivalent to a form of orthogonal (or total) least squares and, thus, (3) A special case of Deming regression on the centered, rescaled vectors taking $\delta = 1$. Orthogonal least squares can be considered as "least-circles regression". – cardinal Jan 3 '12 at 23:41
• @cardinal Very interesting comments! (+1) I believe major axis (minimizing perpendicular distances between reg. line and all the points, à la PCA) or reduced major axis regression, or type II regression as exemplified in the lmodel2 R package by P Legendre, are also relevant here since those techniques are used when it's hard to tell what role (response or predictor) plays each variable or when we want to account for measurement errors. – chl Jan 4 '12 at 10:32
• @chl: (+1) Yes, I believe you are right and the Wikipedia page on total least squares lists several other names for the same procedure, not all of which I am familiar with. It appears to go back to at least R. Frisch, Statistical confluence analysis by means of complete regression systems, Universitetets Økonomiske Instituut, 1934 where it was called diagonal regression. – cardinal Jan 4 '12 at 13:43
• @cardinal I should have been more careful when reading the Wikipedia entry... For future reference, here is a picture taken from Biostatistical Design and Analysis Using R, by M. Logan (Wiley, 2010; Fig. 8.4, p. 174), which summarizes the different approaches, much like Elvis's nice illustrations. – chl Jan 8 '12 at 12:22

Just a brief note on why you see the slope smaller for one regression. Both slopes depend on three numbers: standard deviations of $x$ and $y$ ($s_{x}$ and $s_{y}$), and correlation between $x$ and $y$ ($r$). The regression with $y$ as response has slope $r\frac{s_{y}}{s_{x}}$ and the regression with $x$ as response has slope $r\frac{s_{x}}{s_{y}}$, hence the ratio of the first slope to the reciprocal of the second is equal to $r^2\leq 1$.

So the greater the proportion of variance explained, the closer the slopes obtained from each case. Note that the proportion of variance explained is symmetric and equal to the squared correlation in simple linear regression.

### Regression line is not (always) the same as true relationship

You may have some 'true' causal relationship like

$$y = a + bx + \epsilon$$

but fitted regression lines y ~ x or x ~ y do not mean the same as that causal relationship (even when in practice the expression for one of the regression line may coincide with the expression for the causal 'true' relationship)

### More precise relationship between slopes

For two switched simple linear regressions:

$$Y = a_1 + b_1 X\\X = a_2 + b_2 Y$$

you can relate the slopes as following:

$$b_1 = \rho^2 \frac{1}{b_2} \leq \frac{1}{b_2}$$

So the slopes are not each other inverse.

### Intuition

The reason is that

• Regression lines and correlations do not necessarily correspond one-to-one to a causal relationship.
• Regression lines relate more directly to a conditional probability or best prediction.

You can imagine that the conditional probability relates to the strength of the relationship. Regression lines reflect this and the slopes of the lines may be both shallow when the strength of the relationship is small or both steep when the strength of the relationship is strong. The slopes are not simply each others inverse.

### Example

If two variables $$X$$ and $$Y$$ relate to each other by some (causal) linear relationship $$Y = \text{a little bit of X + a lot of error}$$ Then you can imagine that it would not be good to entirely reverse that relationship in case you wish to express $$X$$ based on a given value of $$Y$$.

$$X = \text{a lot of Y + a little of error}$$

it would be better to also use

$$X = \text{a little bit of Y + a lot of error}$$

See the following example distributions with their respective regression lines. The distributions are multivariate normal with $$\Sigma_{11} \Sigma_{22}=1$$ and $$\Sigma_{12} = \Sigma_{21} = \rho$$ The conditional expected values (what you would get in a linear regression) are

$$\begin{array}{} E(Y|X) &=& \rho X \\ E(X|Y) &=& \rho Y \end{array}$$

and in this case with $$X,Y$$ a multivariate normal distribution, then the marginal distributions are

$$\begin{array}{} Y & \sim & N(\rho X,1-\rho^2) \\ X & \sim & N(\rho Y,1-\rho^2) \end{array}$$

So you can see the variable Y as being a part $$\rho X$$ and a part noise with variance $$1-\rho^2$$. The same is true the other way around.

The larger the correlation coefficient $$\rho$$, the closer the two lines will be. But the lower the correlation, the less strong the relationship, the less steep the lines will be (this is true for both lines Y ~ X and X ~ Y)

• That is a superb explanation. Simple and intuitive – niths4u Dec 20 '20 at 10:39

A simple way to look at this is to note that, if for the true model $$y=\alpha+\beta x+\epsilon$$, you run two regressions:

• $$y=a_{y\sim x}+b_{y\sim x} x$$
• $$x=a_{x\sim y}+b_{x\sim y} y$$

Then we have, using $$b_{y\sim x}=\frac{cov(x,y)}{var(x)}=\frac{cov(x,y)}{var(y)}\frac{var(y)}{var(x)}$$:

$$b_{y\sim x}=b_{x\sim y}\frac{var(y)}{var(x)}$$

So whether you get a steeper slope or not just depends on the ratio $$\frac{var(y)}{var(x)}$$. This ratio is equal to, based on the assumed true model:

$$\frac{var(y)}{var(x)}=\frac{\beta^2 var(x) + var(\epsilon)}{var(x)}$$

You can connect this result with the answers from others, who said that when $$R^2=1$$, it should be the reciprocal. Indeed, $$R^2=1\Rightarrow var(\epsilon) = 0$$, and also, $$b_{y\sim x}=\beta$$ (no estimation error), Hence:

$$R^2=1\Rightarrow b_{y\sim x}=b_{x\sim y}\frac{\beta^2 var(x) + 0}{var(x)}=b_{x\sim y}\beta^2$$

So $$b_{x\sim y}=1/\beta$$

It becomes interesting when there is also noise on your inputs (which we could argue is always the case, no command or observation is ever perfect).

I have built some simulations to observe the phenomenon, based on a simple linear relationship $x = y$, with Gaussian noise on both x and y. I generated the observations as follows (python code):

x = np.linspace(0, 1, n)
y = x

x_o = x + np.random.normal(0, 0.2, n)
y_o = y + np.random.normal(0, 0.2, n)


See the different results (odr here is orthogonal distance regression, i.e. the same as least rectangles regression): All the code is in there:

The goal of a simple linear regression is to come up with the best predictions of the y variable, given values of the x variable. This is a different goal than trying to come up with the best prediction of the x variable, given values of the y variable.

Simple linear regression of y ~ x gives you the 'best' possible model for predicting y given x. Hence, if you fit a model for x ~ y and algebraically inverted it, that model could at its very best do only as well as the model for y ~ x. But inverting a model fit for x ~ y will usually do worse at predicting y given x, compared to the 'optimal' y ~ x model, because the "inverted x ~ y model" was created to fulfill a different objective.

### Illustration

Imagine you have the following dataset: When you run an OLS regression of y ~ x, you come up with the following model

y = 0.167 + 1.5*x


This optimizes predictions of y by making the following predictions, which have associated errors: The OLS regression's predictions are optimal in the sense that the sum of the values in the rightmost column (i.e. the sum of squares) is as small as can be.

When you run an OLS regression of x ~ y, you come up with a different model:

x = -0.07 + 0.64*y


This optimizes predictions of x by making the following predictions, with associated errors. Again, this is optimal in the sense that the sum of the values of the rightmost column are as small as possible (equal to 0.071).

Now, imagine you tried to just invert the first model, y = 0.167 + 1.5*x, using algebra, giving you the model x = -0.11 + 0.67*x.

This would give you the following predictions and associated errors: The sum of the values in the rightmost column is 0.074, which is larger than the corresponding sum from the model you get from regressing x on y, i.e. the x ~ y model. In other words, the "inverted y ~ x model" is doing a worse job at predicting x than the OLS model of x ~ y.