I was asked this in an interview, and I'm curious if (1) my reasoning was correct and (2) if I could have been more precise in my answer. Consider standard OLS with a single predictor:

$$ \mathbf{y} = \mathbf{x} \beta_1 + \boldsymbol{\varepsilon}_1. \tag{1} $$

Now let's say that $\hat{\beta}_1 = 10$. Now imagine we switched the independent and dependent variables, i.e.

$$ \mathbf{x} = \mathbf{y} \beta_2 + \boldsymbol{\varepsilon}_2. \tag{2} $$

Can we say anything about the range of values that $\hat{\beta}_2$ can take? Here is my attempt. The normal equations for each model are:

$$ \begin{aligned} \hat{\beta}_1 &= (\mathbf{x}^{\top} \mathbf{x})^{-1} \mathbf{x}^{\top} \mathbf{y}, \\ \hat{\beta}_2 &= (\mathbf{y}^{\top} \mathbf{y})^{-1} \mathbf{y}^{\top} \mathbf{x}. \end{aligned} \tag{3} $$

Since $\mathbf{x}^{\top} \mathbf{y} = \mathbf{y}^{\top} \mathbf{x}$, then

$$ \begin{aligned} \hat{\beta}_1 &= 10 \\ &\Downarrow \\ (\mathbf{x}^{\top} \mathbf{x}) 10 &= \mathbf{x}^{\top} \mathbf{y} = \mathbf{y}^{\top} \mathbf{x}. \end{aligned} \tag{4} $$

Thus, we can write $\hat{\beta}_2$ as

$$ \begin{aligned} \hat{\beta}_2 &= (\mathbf{y}^{\top} \mathbf{y})^{-1} (\mathbf{x}^{\top} \mathbf{x}) 10 \\ &= \frac{10 \mathbf{x}^{\top} \mathbf{x}}{\mathbf{y}^{\top} \mathbf{y}}. \end{aligned} \tag{5} $$

Since the dot products are both positive, clearly $\hat{\beta}_2 \in [0, \infty)$, and I think we can even say $\hat{\beta}_2 \in (0, \infty)$ if $\mathbf{x}$ is a non-zero predictor. In other words, $\hat{\beta}_2$ must be non-negative. Intuitively, this makes sense. If $\hat{\beta}_1$ is positive, then the relationship between the independent and dependent variables shouldn't change when we switch the regression.

Is this reasoning correct, and can I get a tighter upper bound?


1 Answer 1


Another way to write this in the univariate case that is a little simpler is that $\hat{\beta}_1 = \frac{\text{Cov(x, y)}}{\text{Var}(x)}$. Similarly, $\hat{\beta}_2 = \frac{\text{Cov(x, y)}}{\text{Var}(y)}$. So we know (equivalent to what you've written) $\hat{\beta}_2 = \hat{\beta}_1 \frac{\text{Var}(x)}{\text{Var}(y)}$. It's true that because the variances are positive, $\hat{\beta}_1$ and $\hat{\beta}_2$ have the same sign, but because the variances could be absolutely any positive numbers, there is no upper bound you can place on $\hat{\beta}_2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.