3
$\begingroup$

OLS regression is not symmetric, meaning that it produces different relationships if you flip the dependent and independent variables; however, it would seem odd if the p-values were different and indeed empirical testing shows them to be identical:

import numpy as np
import statsmodels.api as sm
xs = np.random.normal(0, 1, 100)
ys = np.random.normal(0, 1, 100)
sm.OLS(ys, xs).fit().pvalues == sm.OLS(xs, ys).fit().pvalues

What is the proof of this fact?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ What are you looking for here? The easiest way to see that they need to be the same is to think of both forward and backwards tests as tests for zero correlation, which is clearly symmetric. Or do you want to write out the formulas and see they are the same? $\endgroup$
    – Thomas Lumley
    Jun 8 at 7:35
  • $\begingroup$ It's not symmetric, at least not generally. Consider the case where y is constant. $\endgroup$
    – Firebug
    Jun 8 at 11:54
  • 1
    $\begingroup$ But that's just the degenerate case where the $p$-value doesn't exist when you flip the regression, isn't it? In every other case it is symmetric, is it not? $\endgroup$
    – Zaz
    Jun 9 at 22:48
  • $\begingroup$ Yes, it's only symmetric if you impose constraints $\endgroup$
    – Firebug
    Jun 10 at 11:02

1 Answer 1

Reset to default
0
$\begingroup$

This is an over-simplified answer regarding only simple linear regression. I assume you know in advance assumptions of the simple linear regression model I imply and understand notations I use as I am short in time to reply in detail. The Pearson's correlation coefficient is:

$r = \frac{\sum_1^n(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_1^n (x_i - \bar{x})^2\sum_1^n (y_i - \bar{y})^2}}$

It is a fixed quantity in both models. Therefore, $R^2 = r^2$ is also a fixed quantity no matter the equation is $y \sim x$ or $x \sim y$ Assuming linear independence hypothesis.

Therefore, $f = \frac{R^2}{1-R^2}(n-2)$ is a fixed quantity, so is $|t|$ as $f = t^2$.

$P_{value} = 2Pr(T_{n-2} >= |t|)$ will be a fixed quantity no matters what regression equation you choose.

There is definitely a better solution by deriving the t-statistics directly from the formula $t = \frac{b_1}{\sqrt{MSE}}$, but I am short in time to write it down. It is not possible to show what I showed above without going from basic formulas, so my solution is definitely not a good one, but I hope you find it convincing.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.