21
$\begingroup$

Pretty basic question:

What does a normal distribution of residuals from a linear regression mean? In terms of, how does this reflect on my original data from the regression?

I'm totally stumped, thanks guys

$\endgroup$

4 Answers 4

11
$\begingroup$

Linear regression in fact models the conditional expected values of your outcome. That means: if you knew the true values of the regression parameters (say $\beta_0$ and $\beta_1$), given a value of your predictor X, filling that out in the equation $$ E[Y|X] = \beta_0 + \beta_1 X $$ will have you calculate the expected value for $Y$ over all (possible) observations that have this given value for $X$.

However: you don't really expect any single $Y$ value for that given $X$ value to be exactly equal to the (conditional) mean. Not because your model is wrong, but because there are some effects you have not accounted for (e.g. measuring error). So these $Y$ values for a given $X$ values will fluctuate around the mean value (i.e. geometrically: around the point of the regression line for that $X$).

The normality assumption, now, says that the difference between the $Y$s and their matching $E[Y|X]$ follows a normal distribution with mean zero. This means, if you have an $X$ value, then you can sample a $Y$ value by first calculating $\beta_0 + \beta_1 X$ (i.e. again $E[Y|X]$, the point on the regression line), next sampling $\epsilon$ from that normal distribution and adding them: $$ Y'=E[Y|X] + \epsilon $$

In short: this normal distribution represents the variability in your outcome on top of the variability explained by the model.

Note: in most datasets, you don't have multiple $Y$ values for any given $X$ (unless your predictor set is categorical), but this normality goes for the whole population, not just the observations in your dataset.

Note: I've done the reasoning for linear regression with one predictor, but the same goes for more: just replace "line" with "hyperplane" in the above.

$\endgroup$
5
  • $\begingroup$ This is a great explanation! One question though: e being normally distributed would mean that you assume that the most likely values for e are between -1 and +1 (after they were standardized)? So you basically use a normal distribution instead of, say, a poisson distribution, because the normal distribution better models how these values behave in real life? $\endgroup$ Jun 22, 2018 at 20:55
  • $\begingroup$ I'm aware that linear regression can be equivalently derived in multiple ways, one of which is this MLE estimate type motivation. But how would you arrive at a similar reason for normality of residuals, starting from the motivation of trying to minimize sum of squared errors? $\endgroup$ Mar 13, 2021 at 19:16
  • 1
    $\begingroup$ This confusingly implies the errors $\epsilon$ are the residuals in a regression--but they are not. This is an issue because many people fail to distinguish the concepts. $\endgroup$
    – whuber
    Jul 13, 2022 at 13:01
  • $\begingroup$ @whuber: So then the residuals are then the errors in the linear approximation obtained from the sample data, not from the " actual" regression line which is unknowable? $\endgroup$
    – MSIS
    Nov 27, 2023 at 21:22
  • 1
    $\begingroup$ @MSIS That sounds right. $\endgroup$
    – whuber
    Nov 27, 2023 at 22:15
2
$\begingroup$

Normality of the residuals is an assumption of running a linear model. So, if your residuals are normal, it means that your assumption is valid and model inference (confidence intervals, model predictions) should also be valid. It's that simple!

$\endgroup$
4
  • 3
    $\begingroup$ The normality assumption is about unobservable error (hence the need for an assumption), not about observable residuals. $\endgroup$
    – D L Dahly
    Mar 28, 2013 at 15:37
  • 2
    $\begingroup$ Yeah, but you use the residuals to test your assumption about the unobservable error. $\endgroup$
    – wcampbell
    Mar 29, 2013 at 12:38
  • 1
    $\begingroup$ I do not agree that normal residuals guarantee a valid regression model. Suppose you have a circular Gaussian model with X and Y error that are equal. Then the regression line confidence interval is $-\infty \text { to } \infty$. That is hardly the only counter example, there are many more. $\endgroup$
    – Carl
    Aug 6, 2016 at 0:00
  • $\begingroup$ Thank you. Your answer is helpful. $\endgroup$
    – Avv
    Nov 8, 2021 at 20:31
1
$\begingroup$

It could mean a lot or it could mean nothing. If you fit a model to get the highest R-Squared it could mean that you have been foolish. If you fit a model to be parsimonious in that the variables are necessary and needed and care for identifying outliers then you have done a good job. Take a look here for more on this http://www.autobox.com/cms/index.php?option=com_content&view=article&id=175

$\endgroup$
-2
$\begingroup$

In some cases, the assumption that the data is approximately linear allows us to use OLS to minimize the number of observations in the data out that are far from a straight line.

Then the residual is the difference between the true value and fitted value, and we hope this difference is appproximately zero.

But in most real-life cases, the appropriate data is not linear, so we can use some treatment methods or some methods of estimation such as robust tools.

$\endgroup$
1
  • $\begingroup$ This answer doesn't really address the question, presumably why it was downvoted. $\endgroup$
    – Nick Cox
    Jul 3, 2021 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.