In most introductory textbooks (less 'mathematical') simple linear regression model is formulated as equation. i.e.

$$Y_i = \beta_0 + \beta_1 X_{1 i} + \epsilon_i, \quad i =1,n $$ $$ \epsilon_i \sim \mathcal{N}(0, \sigma_\epsilon)$$

Usually this statement presupposes no covariance between $X$ and $\epsilon$.

In some more advanced texts I see more and more regression model description in the kind of alternative form:

$$ Y_i \sim \mathcal{N}(\alpha + \beta X_i, \sigma) $$

And sometimes one can see joint distribution of $X$ and $Y$:

$$ \begin{pmatrix} Y\\X \end{pmatrix} \sim \mathcal{N} \left(\begin{pmatrix} \mu_y \\ \mu_x \end{pmatrix}, \begin{pmatrix} \sigma_y^2 & \rho \sigma_x \sigma_y\\ \rho \sigma_x \sigma_y & \sigma_x^2 \end{pmatrix} \right)$$

where $ \beta = \rho \frac{\sigma_y}{\sigma_x}$.

Is there general convention about the relationship between $\sigma_y$, $\sigma_\epsilon$ and $\sigma$ terms? Are they considered the same or do they need some correction (i.e. for $\rho$ correlation between variables)?

  • 4
    $\begingroup$ @user158565 The are the same provided one relates the parameters appropriately. In particular, the $\sigma$ in the second formulation equals $\sqrt{1-\rho^2}\sigma_y/\sigma_x.$ $\endgroup$ – whuber Jul 16 '19 at 20:22
  • 1
    $\begingroup$ @whuber, thank you! That is exactly what I wanted to be sure about especially concerning the second case. $\endgroup$ – iot Jul 16 '19 at 20:44

Yes there's a relationship between the covariance matrix and the error term variance $\sigma^2$. The two terms $\sigma$ and $\sigma_e$ are redundant, another way of writing it is $\sigma_{y|x}$ which I refer to as the conditional response, provided that the $X$ and $Y$ are indeed related by a linear expectation.

Squared, those terms are the variance of the conditional response $Y - \alpha - \beta X$. $\sigma^2_y$ is the variance of the marginal response. Think about it, if $X$ were random and $Y$ depended perfectly on $X$, then you wouldn't know that $Y$ is not random unless you accounted for $X$. $\sigma^2_y \ge \sigma^2_e$ always.

The formula below shows how marginal and conditional variance are related:

$$ \sigma^2_{y|x} = \sigma^2_y - (\rho^2 \sigma_{x}^2 \sigma_{y}^2) (\sigma^2_x)^{-1}$$

You can see that if $\rho^2$ = 1 (perfect correlation) the whole thing goes to zero.

In fact, it turns out that this relation doesn't in fact depend on the bivariate distribution of $(X,Y)$ being normal. A convenient shorthand my professor used was $X,Y \sim \left( \left( [\mu_x, \mu_y] \right), \Sigma \right)$ meaning the distribution is unspecified beyond its first two moments.

  • 1
    $\begingroup$ The relation requires the joint distribution of $(X,Y)$ to have a linear conditional expectation function. This assumption cannot be dropped! $\endgroup$ – whuber Jul 16 '19 at 22:31
  • $\begingroup$ @AdamO, thank you, why can it be that $\sigma_y > \sigma_e$? if distribution is not normal? $\endgroup$ – iot Jul 17 '19 at 4:52
  • $\begingroup$ @whuber actually, because linear regression is a projection, if you have ANY bivariate distribution $(X,Y)$ with (finite) covariance matrix $\Sigma$, the variance of the residuals is $\Sigma_{2,2} - \Sigma_{2,1}^2/\Sigma_{1,1}$. The residual distribution may be non-normal. The only way that normality (of response) AND linearity is important is so that we can say that the residuals have a normal distribution. $\endgroup$ – AdamO Jul 17 '19 at 15:27
  • 2
    $\begingroup$ I think you might be correct provided we are very clear about the meaning of your terms, Adam: it's important to distinguish the conditional distribution of $Y$ from the residuals in the linear regression of $Y$ against $X.$ Indeed, in makes little sense even to speak of "the variance of the residuals" unless the residuals are homoscedastic. Linearity of the conditional expectation is important, too, for otherwise this regression model is inappropriate. $\endgroup$ – whuber Jul 17 '19 at 16:54
  • 2
    $\begingroup$ @whuber ah that is an important distinction, and seems squarely inline with the OP's question, so I made a little note/edit at the top. This conversation may be useful for those who are considering a more general case. Thanks. $\endgroup$ – AdamO Jul 17 '19 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.