least square estimator of regression x onto y [duplicate]

Question

I've been reading linear regression and least square estimator. Suppose we have i.i.d data $(x_1, y_q), (x_2, y_2), ..., (x_n,y_n)$ such that we use a linear regression model $y_i = \beta x_i + \epsilon_i$ and learnt the fact that we often derive least square estimator $\hat{\beta} = \frac{\sum_{i = 1}^{n}x_iy_i}{\sum_{i = 1}^{n}x_i^2}$.

However I wonder what would be the effect if we do regression of x onto y. Would we then be able to use the least square estimate for the regression of x onto y to estimate of $\frac{1}{\beta}$?

How do we decide whether this is a good estimate in the first place?

Answer 1

To prove that the reverse regression is not a good estimator for $1/\beta$, recall that OLS is generally consistent (when regressing $y$ on $x$) for $cov(x,y)/var(x)$. Correspondingly, it is consistent for $cov(x,y)/var(y)$ when regressing $x$ on $y$.

When the relationship between error and regressor is (essentially, predeterminedness) is such that $\beta=cov(x,y)/var(x)$, to have that $cov(x,y)/var(y)=1/\beta$ would require that $$ cov(x,y)/var(y)=var(x)/cov(x,y), $$ and there is no reason to expect this to hold in general.

In fact, the condition could be reexpressed as $$ \frac{cov(x,y)^2}{var(y)var(x)}=1, $$ which is the limiting case of the Cauchy-Schwarz inequality, which is known to only obtain if the random variables in question are multiples of each other.

In that case, we have, say, $y=\beta x$, so that $$ \frac{cov(x,y)}{var(x)}=\beta \cdot var(x)/var(x)=\beta $$ and $$ \frac{cov(x,y)}{var(y)}=\frac{\beta \cdot var(x)}{\beta ^2var(x)}=\frac{1}{\beta } $$

Here is a little graphical illustration (where you'd want to read the cases of regressing $x$ on $y$ rotating the plot counterclockwise by 90 degrees):

library(mvtnorm)
n <- 10000
cov.xy <- 0.5
var.y <- 1
var.x <- 4
beta <- cov.xy/var.x
dat <- rmvnorm(n, mean = rep(0,2), sigma = matrix(c(var.y, cov.xy, cov.xy, var.x), ncol=2))

y <- dat[,1]
x <- dat[,2]

par(mfrow=c(1,2))
plot(x, y, pch=19, cex=0.2, col="lightgreen")
abline(lm(y~x),lwd=2, col="lightgreen")          # a regression of y on x
abline(a=0, b=beta, lwd=2, col="green")          # what OLS of y on x is consistent for

plot(y, x, pch=19, cex=0.2, col="lightblue")
abline(lm(x~y), lwd=2, col="lightblue")          # a regression of x on y
abline(a=0, cov.xy/var.y, lwd=2, col="darkblue") # what OLS of x on y is consistent for
abline(a=0, b=1/beta, lwd=2, col="red")          # what OLS of x on y is NOT consistent for

Answer 2

No, in general you will obtain a different line wih ordinary least squares if you interchange x and y. You can easily check this with your formula by interchanging x and y and comparing it to $1/\beta$. The reaaon for this descrepancy is that ordinary least squares is not about fitting a line through points, but about prediction and thus assumes a specific role of the variables: x is "predictor", y is "response".

If your problem is actually about fitting a line through points, you should consider "orthogonal least squares", which is a symmetric approach and has (for straight lines) two equivalent solutions:

1. the right singular vector $\vec{v}_1$ corresponding to the largest singular value $s_1\geq\ldots\geq s_n$ in the singular value decomposition (SVD) $Q=USV^T$ of the matrix built from the centered data points $$Q^T = (\vec{q}_1,\ldots,\vec{q}_n) \quad\mbox{ with }\quad \vec{q}_i = \vec{x}_i - \vec{a}$$

2. the eigenvector corresponding to the largest eigenvalue of $Q^TQ$. $Q^TQ$ is identical to the scatter matrix, or $(n-1)$ times the covariance matrix of the data points $\vec{x}_1,\ldots,\vec{x}_n$. Thus, this vector is simply the principal component obtained from a principal component analysis (PCA)

Note that orthogonal least squares also yields a reasonable result when the points happen to fall on (or around) a vertical line.

Reference:

H. Späth: "Orthogonal least squares fitting with linear manifolds." Numerische Mathematik 48 (1986), pp. 441–445.