Regression in linear equation vs. distribution form

Question

In most introductory textbooks (less 'mathematical') simple linear regression model is formulated as equation. i.e.

$$Y_i = \beta_0 + \beta_1 X_{1 i} + \epsilon_i, \quad i =1,n $$ $$ \epsilon_i \sim \mathcal{N}(0, \sigma_\epsilon)$$

Usually this statement presupposes no covariance between $X$ and $\epsilon$.

In some more advanced texts I see more and more regression model description in the kind of alternative form:

$$ Y_i \sim \mathcal{N}(\alpha + \beta X_i, \sigma) $$

And sometimes one can see joint distribution of $X$ and $Y$:

$$ \begin{pmatrix} Y\\X \end{pmatrix} \sim \mathcal{N} \left(\begin{pmatrix} \mu_y \\ \mu_x \end{pmatrix}, \begin{pmatrix} \sigma_y^2 & \rho \sigma_x \sigma_y\\ \rho \sigma_x \sigma_y & \sigma_x^2 \end{pmatrix} \right)$$

where $ \beta = \rho \frac{\sigma_y}{\sigma_x}$.

Is there general convention about the relationship between $\sigma_y$, $\sigma_\epsilon$ and $\sigma$ terms? Are they considered the same or do they need some correction (i.e. for $\rho$ correlation between variables)?

Answer 1

Yes there's a relationship between the covariance matrix and the error term variance $\sigma^2$. The two terms $\sigma$ and $\sigma_e$ are redundant, another way of writing it is $\sigma_{y|x}$ which I refer to as the conditional response, provided that the $X$ and $Y$ are indeed related by a linear expectation.

Squared, those terms are the variance of the conditional response $Y - \alpha - \beta X$. $\sigma^2_y$ is the variance of the marginal response. Think about it, if $X$ were random and $Y$ depended perfectly on $X$, then you wouldn't know that $Y$ is not random unless you accounted for $X$. $\sigma^2_y \ge \sigma^2_e$ always.

The formula below shows how marginal and conditional variance are related:

$$ \sigma^2_{y|x} = \sigma^2_y - (\rho^2 \sigma_{x}^2 \sigma_{y}^2) (\sigma^2_x)^{-1}$$

You can see that if $\rho^2$ = 1 (perfect correlation) the whole thing goes to zero.

In fact, it turns out that this relation doesn't in fact depend on the bivariate distribution of $(X,Y)$ being normal. A convenient shorthand my professor used was $X,Y \sim \left( \left( [\mu_x, \mu_y] \right), \Sigma \right)$ meaning the distribution is unspecified beyond its first two moments.