We have the model for the linear regression

$$y=X\beta + \varepsilon.$$

Why is $\text{Var}(\varepsilon)=\sigma^2$?

Given the fact that we don't know the distribution of the error term, $\varepsilon$, how can we make assumption on the variance?

up vote 4 down vote accepted

In ordinary linear regression we do know the distribution of the error term $\varepsilon$, up to the single unknown parameter $\sigma^2$. Namely, our model is that the errors are drawn iid from the distribution $$\varepsilon_i \sim \mathcal N(0,\sigma^2).$$

We then estimate $\sigma^2$ along with the unknown $\beta$ coefficients.

  • I see, thank you very much Jonathan. What's the reasoning behind setting $E(\varepsilon)=0$ ? – Chris Jan 26 '13 at 0:40
  • 2
    The premise of the model is that $E(y) = X\beta$. But $E(y) = E(X\beta + \varepsilon) = X\beta+E(\varepsilon)$ (since $X\beta$ is not random), so we can only have $E(y) = X\beta$ if $E(\varepsilon)=0$. – Jonathan Christensen Jan 26 '13 at 0:47
  • Ahh, I see. Thank you so much! – Chris Jan 26 '13 at 0:53
  • What do you mean by: "We then estaimte $\sigma^2$ along with the unkown coefficients"? In OLS, I only obtain values for the $\beta$ coefficients. I do not obtain a value $\sigma^2$. – Julian Dec 29 '16 at 12:27
  • You should not always make the assumption the error term is normally distributed. You must exert caution when making assumptions about the error term. There is a substantial body of work in non-parametric modeling which offers convincing evidence to suggest it can be dangerous in estimation. – JuliusBilly Apr 6 at 17:09

We should not make this assumption uncritically.

The error term $\varepsilon_i$ conditional on a particular $X$ value $X_i$, like any random variable, has a variance, usually written $\sigma_i^2$. There is no assumption here, it is just notation for that variance.

However, one of the assumptions of classical linear regression is that the error terms conditional on different $X$ values all have the same variance, that is, for any $X_i$ and $X_j$, $\sigma_i^2 = \sigma_j^2$. This assumption, known as homoscedasticity, may or may not be met for a particular model applied to a particular population. Before drawing conclusions from ordinary least squares (OLS) regression it is good practice to apply appropriate tests (or at least inspection of residuals) to assess whether this assumption is met. Where the assumption is met we are justified in using a common symbol, usually $\sigma^2$, for the common variance of the error terms.

Where the assumption is not met, that is, where there is heteroscedasticity, OLS regression is liable to give biased estimates of the variances of the regression coefficients. In that case weighted least squares is used to correct for the heteroscedasticity. Properly used, this has the effect of transforming the model in such a way that homoscedasticity is restored.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.