We know beta for univariate OLS; what can we say about beta after switching the independent and dependent variables?

Question

I was asked this in an interview, and I'm curious if (1) my reasoning was correct and (2) if I could have been more precise in my answer. Consider standard OLS with a single predictor:

$$ \mathbf{y} = \mathbf{x} \beta_1 + \boldsymbol{\varepsilon}_1. \tag{1} $$

Now let's say that $\hat{\beta}_1 = 10$. Now imagine we switched the independent and dependent variables, i.e.

$$ \mathbf{x} = \mathbf{y} \beta_2 + \boldsymbol{\varepsilon}_2. \tag{2} $$

Can we say anything about the range of values that $\hat{\beta}_2$ can take? Here is my attempt. The normal equations for each model are:

$$ \begin{aligned} \hat{\beta}_1 &= (\mathbf{x}^{\top} \mathbf{x})^{-1} \mathbf{x}^{\top} \mathbf{y}, \\ \hat{\beta}_2 &= (\mathbf{y}^{\top} \mathbf{y})^{-1} \mathbf{y}^{\top} \mathbf{x}. \end{aligned} \tag{3} $$

Since $\mathbf{x}^{\top} \mathbf{y} = \mathbf{y}^{\top} \mathbf{x}$, then

$$ \begin{aligned} \hat{\beta}_1 &= 10 \\ &\Downarrow \\ (\mathbf{x}^{\top} \mathbf{x}) 10 &= \mathbf{x}^{\top} \mathbf{y} = \mathbf{y}^{\top} \mathbf{x}. \end{aligned} \tag{4} $$

Thus, we can write $\hat{\beta}_2$ as

$$ \begin{aligned} \hat{\beta}_2 &= (\mathbf{y}^{\top} \mathbf{y})^{-1} (\mathbf{x}^{\top} \mathbf{x}) 10 \\ &= \frac{10 \mathbf{x}^{\top} \mathbf{x}}{\mathbf{y}^{\top} \mathbf{y}}. \end{aligned} \tag{5} $$

Since the dot products are both positive, clearly $\hat{\beta}_2 \in [0, \infty)$, and I think we can even say $\hat{\beta}_2 \in (0, \infty)$ if $\mathbf{x}$ is a non-zero predictor. In other words, $\hat{\beta}_2$ must be non-negative. Intuitively, this makes sense. If $\hat{\beta}_1$ is positive, then the relationship between the independent and dependent variables shouldn't change when we switch the regression.

Is this reasoning correct, and can I get a tighter upper bound?

Answer 1

Another way to write this in the univariate case that is a little simpler is that $\hat{\beta}_1 = \frac{\text{Cov(x, y)}}{\text{Var}(x)}$. Similarly, $\hat{\beta}_2 = \frac{\text{Cov(x, y)}}{\text{Var}(y)}$. So we know (equivalent to what you've written) $\hat{\beta}_2 = \hat{\beta}_1 \frac{\text{Var}(x)}{\text{Var}(y)}$. It's true that because the variances are positive, $\hat{\beta}_1$ and $\hat{\beta}_2$ have the same sign, but because the variances could be absolutely any positive numbers, there is no upper bound you can place on $\hat{\beta}_2$.