5
$\begingroup$

I'm confused with how to interpret the results of an OLS regression when you switch the response and predictor variables and both intercepts are positive. More concretely

$$ y = \beta_{x}x + \alpha_{x} \\ x = \beta_{y}y + \alpha_{y} $$

then the OLS estimators are

$$ \hat{\alpha_{x}} = \bar{y} - \hat{\beta_{x}}\bar{x} \\ \hat{\beta_{x}} = \frac{Cov(x,y)}{\sigma^{2}_{x}} $$

and

$$ \hat{\alpha_{y}} = \bar{x} - \hat{\beta_{y}}\bar{y} \\ \hat{\beta_{y}} = \frac{Cov(x,y)}{\sigma^{2}_{y}} $$

respectively. Then substituting in for $\beta$ in each equation we get

$$ \hat{\alpha_{x}} = \bar{y} - \frac{Cov(x,y)}{\sigma^{2}_{x}}\bar{x} \\ \hat{\alpha_{y}} = \bar{x} - \frac{Cov(x,y)}{\sigma^{2}_{y}}\bar{y} \\ $$

If we want both $\hat{\alpha_{x}} > 0 $ & $\hat{\alpha_{y}} > 0$, it's straightforward enough to satisfy this in the above equations. For example choosing

$$ \begin{align} \bar{y} &= 2 \\ \bar{x} &= 1 \\ Cov(x,y) &= 1.9 \\ \sigma_{x}^{2} &= 1 \\ \sigma_{y}^{2} &= 9 \\ \end{align} $$

from which we obtain

$$ \hat{\alpha_{x}} = \bar{y} - \frac{Cov(x,y)}{\sigma^{2}_{x}}\hat{x} = 2 - \frac{1.9}{1} \cdot 1 = 0.1 $$

and

$$ \hat{\alpha_{y}} = \bar{x} - \frac{Cov(x,y)}{\sigma^{2}_{y}}\bar{y} = 1 - \frac{1.9}{9} \cdot 2 \approx 0.58 $$

this is confirmed using some simulated data in $R$.

set.seed(42)
N <- 1000
eps1 <- rnorm(N)
eps2 <- rnorm(N)
X <- 1 + eps1
Y <- 2 + 1.9*eps1 + sqrt(9-1.9^2)*eps2

Regressing $Y$ on $X$ we get

summary(lm(formula = Y ~ X))

Call:
lm(formula = Y ~ X)

Residuals:
    Min      1Q  Median      3Q     Max 
-6.7849 -1.5295 -0.0193  1.5388  8.3292 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.06545    0.10101   0.648    0.517    
X            1.92279    0.07228  26.602   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.29 on 998 degrees of freedom
Multiple R-squared:  0.4149,    Adjusted R-squared:  0.4143 
F-statistic: 707.7 on 1 and 998 DF,  p-value: < 2.2e-16

And vis versa

summary(lm(formula = X ~ Y))

Call:
lm(formula = X ~ Y)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.62005 -0.52981  0.01435  0.53093  2.44786 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.555875   0.028912   19.23   <2e-16 ***
Y           0.215776   0.008111   26.60   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7672 on 998 degrees of freedom
Multiple R-squared:  0.4149,    Adjusted R-squared:  0.4143 
F-statistic: 707.7 on 1 and 998 DF,  p-value: < 2.2e-16

However I find this result somewhat counterintuitive.

What is the intuition of both intercepts in an OLS regressions being greater than 0 when x and y are exchanged as predictor and response variables. Does anyone have any good visuals or explanations of the subset of dependency structures that would lead to this kind of result?

$\endgroup$
0
5
$\begingroup$

Due to regression toward the mean, it would (for example) be expected to occur fairly readily if the means of $x$ and $y$ were positive (with $x$ and $y$ positively correlated), and the total least squares line went sufficiently close to 0 (where "sufficiently close" depends on how strong the relationship is as well as the scales* and the location of the mean point - the weaker the relationship the further the intercept on the TLS line could stray from 0 without causing either least-squares intercept to go negative)

Here's an example I constructed following exactly that recipe:

enter image description here

The green line is obtained from principal component analysis. The red line is the least squares line for the regression of y on x, while the blue line is for the regression of x on y.

Regression toward the mean pulls the red line clockwise and the blue line anti-clockwise from the green one, which means the y-intercept for the red line will be above the green one and that for the blue line will be below the green one. The green line will fall to one side of the origin; you just need the origin to be relatively close to it so that the regression line that's on the same side of it as the origin is will lay the opposite side of the origin point, in which case it will have its least--squares-line intercept being positive.

* on reflection, it may have been better to frame it in terms of a regression based on PCA of the correlation matrix (PCA on standardized variates), but I'll leave it as is for now.

$\endgroup$
2
  • $\begingroup$ This is an interesting way to think about it, thanks. However I'm a bit confused by your footnote, I don't really understand how standardizing $x$ and $y$ would conceptually change the explanation? $\endgroup$
    – mgilbert
    Sep 25 '15 at 11:16
  • $\begingroup$ It doesn't change the form of the explanation at all; it just makes the plot look more "intuitive"; the green line would sit nearer the middle of the two ordinary regression lines. The same effect could be achieved by rescaling my $y$ to have the same variance as $x$ and sticking with ordinary orthogonal regression. $\endgroup$
    – Glen_b
    Sep 25 '15 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.