# Intuition of both OLS intercept being positive when response and predictor variables are exchanged

I'm confused with how to interpret the results of an OLS regression when you switch the response and predictor variables and both intercepts are positive. More concretely

$$y = \beta_{x}x + \alpha_{x} \\ x = \beta_{y}y + \alpha_{y}$$

then the OLS estimators are

$$\hat{\alpha_{x}} = \bar{y} - \hat{\beta_{x}}\bar{x} \\ \hat{\beta_{x}} = \frac{Cov(x,y)}{\sigma^{2}_{x}}$$

and

$$\hat{\alpha_{y}} = \bar{x} - \hat{\beta_{y}}\bar{y} \\ \hat{\beta_{y}} = \frac{Cov(x,y)}{\sigma^{2}_{y}}$$

respectively. Then substituting in for $\beta$ in each equation we get

$$\hat{\alpha_{x}} = \bar{y} - \frac{Cov(x,y)}{\sigma^{2}_{x}}\bar{x} \\ \hat{\alpha_{y}} = \bar{x} - \frac{Cov(x,y)}{\sigma^{2}_{y}}\bar{y} \\$$

If we want both $\hat{\alpha_{x}} > 0$ & $\hat{\alpha_{y}} > 0$, it's straightforward enough to satisfy this in the above equations. For example choosing

\begin{align} \bar{y} &= 2 \\ \bar{x} &= 1 \\ Cov(x,y) &= 1.9 \\ \sigma_{x}^{2} &= 1 \\ \sigma_{y}^{2} &= 9 \\ \end{align}

from which we obtain

$$\hat{\alpha_{x}} = \bar{y} - \frac{Cov(x,y)}{\sigma^{2}_{x}}\hat{x} = 2 - \frac{1.9}{1} \cdot 1 = 0.1$$

and

$$\hat{\alpha_{y}} = \bar{x} - \frac{Cov(x,y)}{\sigma^{2}_{y}}\bar{y} = 1 - \frac{1.9}{9} \cdot 2 \approx 0.58$$

this is confirmed using some simulated data in $R$.

set.seed(42)
N <- 1000
eps1 <- rnorm(N)
eps2 <- rnorm(N)
X <- 1 + eps1
Y <- 2 + 1.9*eps1 + sqrt(9-1.9^2)*eps2


Regressing $Y$ on $X$ we get

summary(lm(formula = Y ~ X))

Call:
lm(formula = Y ~ X)

Residuals:
Min      1Q  Median      3Q     Max
-6.7849 -1.5295 -0.0193  1.5388  8.3292

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.06545    0.10101   0.648    0.517
X            1.92279    0.07228  26.602   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.29 on 998 degrees of freedom
Multiple R-squared:  0.4149,    Adjusted R-squared:  0.4143
F-statistic: 707.7 on 1 and 998 DF,  p-value: < 2.2e-16


And vis versa

summary(lm(formula = X ~ Y))

Call:
lm(formula = X ~ Y)

Residuals:
Min       1Q   Median       3Q      Max
-2.62005 -0.52981  0.01435  0.53093  2.44786

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.555875   0.028912   19.23   <2e-16 ***
Y           0.215776   0.008111   26.60   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7672 on 998 degrees of freedom
Multiple R-squared:  0.4149,    Adjusted R-squared:  0.4143
F-statistic: 707.7 on 1 and 998 DF,  p-value: < 2.2e-16


However I find this result somewhat counterintuitive.

What is the intuition of both intercepts in an OLS regressions being greater than 0 when x and y are exchanged as predictor and response variables. Does anyone have any good visuals or explanations of the subset of dependency structures that would lead to this kind of result?

Due to regression toward the mean, it would (for example) be expected to occur fairly readily if the means of $x$ and $y$ were positive (with $x$ and $y$ positively correlated), and the total least squares line went sufficiently close to 0 (where "sufficiently close" depends on how strong the relationship is as well as the scales* and the location of the mean point - the weaker the relationship the further the intercept on the TLS line could stray from 0 without causing either least-squares intercept to go negative)
• This is an interesting way to think about it, thanks. However I'm a bit confused by your footnote, I don't really understand how standardizing $x$ and $y$ would conceptually change the explanation? Sep 25 '15 at 11:16
• It doesn't change the form of the explanation at all; it just makes the plot look more "intuitive"; the green line would sit nearer the middle of the two ordinary regression lines. The same effect could be achieved by rescaling my $y$ to have the same variance as $x$ and sticking with ordinary orthogonal regression. Sep 25 '15 at 12:24