I am trying to know the assumptions of linear regression (LR). I understand linear regression needs the relationship between the independent and dependent variables to be linear, but LR also assumes the input residuals to be normally distributed. Please help me understand this.
$\begingroup$
$\endgroup$
4
-
2$\begingroup$ Related: ANOVA assumption normality/normal distribution of residuals. $\endgroup$– gung - Reinstate MonicaSep 4, 2019 at 20:40
-
2$\begingroup$ Residuals and errors are distinct things. Errors are unobservable; residuals may be considered estimates of errors but they are not inputs; they're outputs. The assumption of normality is used when deriving the usual tests, confidence intervals and prediction intervals; if you don't use those you don't necessarily require normality - but it's useful to keep in mind that the least squares estimates are optimal (in a particular sense) in that normality case, but will be less-than-optimal under other distributional assumptions (though still optimal in the best-linear-unbiased sense). $\endgroup$– Glen_bSep 4, 2019 at 23:49
-
$\begingroup$ I would assume the OP actually means the error term in the model rather than the residuals. Of course I do agree with $Glen_b s point. But another point that should be mentioned here is that linear regression does not assume normally distributed error terms. It only assumes that error terms are uncorrelated with 0 mean & common variance. It is only when you want to use least squares for the solution & want that to be a maximum likelihood solution that the normal distribution enters.. $\endgroup$– Michael R. ChernickSep 5, 2019 at 18:40
-
$\begingroup$ Even if the error distribution is not normally distributed by the Gauss-Markov theorem least squares has the best linear unbiased property. $\endgroup$– Michael R. ChernickSep 5, 2019 at 18:43
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Take a standard normal random variable.
$$z \sim \mathcal{N}(0,1)$$
It has the property that if I add $\mu$ to $z$ then
$$(\mu+ z) \sim\mathcal{N}(\mu,1)$$
So now, consider linear regression. Linear regression states that the mean of the outcome conditioned on the covariates is a linear combination of the covariates. Let $\mu(x) = \beta_0 + \sum_i x_i \beta_i$. Then,
$$ y\vert x \sim \mathcal{N}\left(\mu(x), \sigma^2 \right) $$
Going backwards now, that means that
$$ (y\vert x - \mu(x) ) \sim \mathcal{N}(0,\sigma^2) $$
So as you can see, as a consequence of assuming that the likelihood of our model is normal, the residuals (that is, the data minus the predicted conditional mean) is normally distributed with mean 0 and some constant variance.
answered Sep 4, 2019 at 19:58
-
1$\begingroup$ They posted question is not whether they're necessary but why they're assumed at all. $\endgroup$– Glen_bSep 4, 2019 at 23:50
$\begingroup$
$\endgroup$
3
Well, first thing first - Linear regression does not assume the residuals are normally distributed. Linear regression is an approach to find the best fitted linear model to the observed data. Now, we, the observers, look at the residuals and we can assume that these are distributed normally. Why? well, for once, if it does, it gives us a lot of power - from finding confidence intervals and doing hypothesis testing to even more application and tools that can only be performed under the assumption that the residuals are distributed from some distribution. The use of normal distribution is cause this probability distribution is very well understood, it's properties are usually easily computed, and it's geometric interpretation comes up and describe many things in the measurable world.
-
$\begingroup$ Hard to understand why this answer got a -1 without any comment. Must be from a true expert... $\endgroup$ Sep 4, 2019 at 19:44
-
3$\begingroup$ I agree with this answer, but I think it's less clear & complete than it could be (I'm not the one who downvoted). If I were going to answer/upvote an answer it would clearly separate what we can say about linear regression without the assumption of Normality (Gauss-Markov theorem, blah blah blah) and what we get additionally by assuming Normality (inferential procedures - confidence intervals etc.) $\endgroup$ Sep 4, 2019 at 19:50
-
1$\begingroup$ The assumption of normally distributed residuals is used to claim certain optimal properties of least squares (e.g. maximum likelihood property).. As this answer mentions it is not a necessary assumption. In fact least squares is best linear unbiased without the assumption of normality.only assuming uncorrelated residuals with 0 mean & common variance (the Gauss-Markov property). $\endgroup$ Sep 4, 2019 at 20:16