# least square estimator of regression x onto y [duplicate]

I've been reading linear regression and least square estimator. Suppose we have i.i.d data $$(x_1, y_q), (x_2, y_2), ..., (x_n,y_n)$$ such that we use a linear regression model $$y_i = \beta x_i + \epsilon_i$$ and learnt the fact that we often derive least square estimator $$\hat{\beta} = \frac{\sum_{i = 1}^{n}x_iy_i}{\sum_{i = 1}^{n}x_i^2}$$.

However I wonder what would be the effect if we do regression of x onto y. Would we then be able to use the least square estimate for the regression of x onto y to estimate of $$\frac{1}{\beta}$$?

How do we decide whether this is a good estimate in the first place?

• Are you deliberately missing a constant term in your regression? Or are the means zero? Commented Oct 3, 2020 at 7:48

To prove that the reverse regression is not a good estimator for $$1/\beta$$, recall that OLS is generally consistent (when regressing $$y$$ on $$x$$) for $$cov(x,y)/var(x)$$. Correspondingly, it is consistent for $$cov(x,y)/var(y)$$ when regressing $$x$$ on $$y$$.

When the relationship between error and regressor is (essentially, predeterminedness) is such that $$\beta=cov(x,y)/var(x)$$, to have that $$cov(x,y)/var(y)=1/\beta$$ would require that $$cov(x,y)/var(y)=var(x)/cov(x,y),$$ and there is no reason to expect this to hold in general.

In fact, the condition could be reexpressed as $$\frac{cov(x,y)^2}{var(y)var(x)}=1,$$ which is the limiting case of the Cauchy-Schwarz inequality, which is known to only obtain if the random variables in question are multiples of each other.

In that case, we have, say, $$y=\beta x$$, so that $$\frac{cov(x,y)}{var(x)}=\beta \cdot var(x)/var(x)=\beta$$ and $$\frac{cov(x,y)}{var(y)}=\frac{\beta \cdot var(x)}{\beta ^2var(x)}=\frac{1}{\beta }$$

Here is a little graphical illustration (where you'd want to read the cases of regressing $$x$$ on $$y$$ rotating the plot counterclockwise by 90 degrees):

library(mvtnorm)
n <- 10000
cov.xy <- 0.5
var.y <- 1
var.x <- 4
beta <- cov.xy/var.x
dat <- rmvnorm(n, mean = rep(0,2), sigma = matrix(c(var.y, cov.xy, cov.xy, var.x), ncol=2))

y <- dat[,1]
x <- dat[,2]

par(mfrow=c(1,2))
plot(x, y, pch=19, cex=0.2, col="lightgreen")
abline(lm(y~x),lwd=2, col="lightgreen")          # a regression of y on x
abline(a=0, b=beta, lwd=2, col="green")          # what OLS of y on x is consistent for

plot(y, x, pch=19, cex=0.2, col="lightblue")
abline(lm(x~y), lwd=2, col="lightblue")          # a regression of x on y
abline(a=0, cov.xy/var.y, lwd=2, col="darkblue") # what OLS of x on y is consistent for
abline(a=0, b=1/beta, lwd=2, col="red")          # what OLS of x on y is NOT consistent for


• how does the fact that OLS is consistent with regressing x onto y has to do with it is not a good estimate? Commented Oct 4, 2020 at 17:17
• Consistency is a reasonably well-accepted (frequentist) criterion for a good estimator. And my post attempts to demonstrate that the coefficient of a regression of $x$ and $y$ will generally not be consistent for $1/\beta$ if the coefficient of a regression of $y$ on $x$ is consistent for $\beta$. Commented Oct 5, 2020 at 6:39

No, in general you will obtain a different line wih ordinary least squares if you interchange x and y. You can easily check this with your formula by interchanging x and y and comparing it to $$1/\beta$$. The reaaon for this descrepancy is that ordinary least squares is not about fitting a line through points, but about prediction and thus assumes a specific role of the variables: x is "predictor", y is "response".

If your problem is actually about fitting a line through points, you should consider "orthogonal least squares", which is a symmetric approach and has (for straight lines) two equivalent solutions:

1. the right singular vector $$\vec{v}_1$$ corresponding to the largest singular value $$s_1\geq\ldots\geq s_n$$ in the singular value decomposition (SVD) $$Q=USV^T$$ of the matrix built from the centered data points $$Q^T = (\vec{q}_1,\ldots,\vec{q}_n) \quad\mbox{ with }\quad \vec{q}_i = \vec{x}_i - \vec{a}$$

2. the eigenvector corresponding to the largest eigenvalue of $$Q^TQ$$. $$Q^TQ$$ is identical to the scatter matrix, or $$(n-1)$$ times the covariance matrix of the data points $$\vec{x}_1,\ldots,\vec{x}_n$$. Thus, this vector is simply the principal component obtained from a principal component analysis (PCA)

Note that orthogonal least squares also yields a reasonable result when the points happen to fall on (or around) a vertical line.

Reference:

H. Späth: "Orthogonal least squares fitting with linear manifolds." Numerische Mathematik 48 (1986), pp. 441–445.

• Thanks! I wonder whether there's a way to prove that it is not a good estimator for $\frac{1}{\beta}$? Commented Oct 2, 2020 at 11:04