13
$\begingroup$

In simple linear regression one often wants to verify if certain assumptions are met to be able to do inference (e.g. residuals are normally distributed).

Is it reasonable to check the assumptions by checking if the fitted values are normally distributed?

$\endgroup$

1 Answer 1

20
$\begingroup$

Why are diagnostics based on residuals?

Because many of the assumptions relate to the conditional distribution of $Y$, not its unconditional distribution. That's equivalent to an assumption on the errors, which we estimate by the residuals.

In simple linear regression one often wants to verify if certain assumptions are met to be able to do inference (e.g. residuals are normally distributed).

The actual normality assumption isn't about the residuals but about the error term. The nearest thing to those that you have are the residuals, which is why we check them.

Is it reasonable to check check the assumptions by checking if the fitted values are normally distributed?

No. The distribution of fitted values depends on the pattern of the $x$'s. It doesn't tell you much at all about the assumptions.

For example, I just ran a regression on simulated data, for which all the assumptions were correctly specified. For example the normality of the errors was satisfied. Here's what happens when we try to check the normality of the fitted values:

normality diagnostics on fitteds

They're clearly non-normal; in fact they look bimodal. Why? Well, because the distribution of fitted values depends on the pattern of the $x$'s. The errors were normal, but the fitted values might be almost anything.

Another thing people often check (much more often, in fact) is normality of the $y$s... but unconditionally on $x$; again, this depends on the pattern of $x$s, and so doesn't tell you much about the actual assumptions. Again, I've generated some data where the assumptions all hold; here's what happens when we try to check normality of the unconditional $y$ values:

normality diagnostics on raw y-values

Again, the non-normality we see here (the y's are skew) is not related to the conditional normality of the $y$s.

In fact I have a textbook next to me right now that discusses this distinction (between the conditional distribution and unconditional distribution of $Y$) - that is, it explains in an early chapter why just looking at the distribution of the $y$'s isn't right $-$ and then in subsequent chapters repeatedly checks the normality assumption by looking at the distribution of the $y$ values $-$ without considering the impact of the $x$'s $-$ to assess the suitability of the assumptions (another thing it usually does is to just look at histograms to make that assessment, but that's a whole other problem).


What are the assumptions, how do we check them and when do we need to make them?

  • The $x$'s may be treated as fixed (observed without error). We don't generally try to check this diagnostically (but we should have a good idea whether it's true).

  • The relationship between $E(Y)$ and the $x$ in the model is correctly specified (e.g., linear). If we subtract the best fitting linear model, there should be no remaining pattern in the relationship between the mean of the residuals and $x$.

  • Constant variance (i.e., $\text{Var}(Y|x)$ doesn't depend on $x$. The spread of the errors is constant; it might be checked by looking at the spread of the residuals against $x$, or by checking some function of the squared residuals against $x$ and checking for changes in the average (e.g., such functions as the log, or the square root. R uses the fourth root of the squared residuals).

  • Conditional independence/independence of errors. Particular forms of dependence can be checked for (e.g., serial correlation). If you can't anticipate the form of dependence, it's a little hard to check.

  • Normality the conditional distribution of $Y$/normality of errors. Can be checked, for example, by doing a Q-Q plot of residuals.

(There are actually some other assumptions I haven't mentioned, such as additive errors, that the errors have zero mean, and so on.)

If you're only interested in estimating the fit of the least squares line and not in say standard errors, you don't need to make most of these assumptions. For example, the distribution of errors affects inference (tests and intervals), and it can affect the efficiency of the estimate, but the LS line is still best linear unbiased for example; so unless the distribution is so badly non-normal that all linear estimators are bad, it's not necessarily much of an issue if the assumptions about the error term don't hold.

$\endgroup$
7
  • $\begingroup$ I added some diagrams to my answer. $\endgroup$
    – Glen_b
    Nov 11, 2013 at 0:34
  • 2
    $\begingroup$ This is a great answer. If you want more, I cover somewhat similar territory here: What if residuals are normally distributed but Y is not? $\endgroup$ Nov 11, 2013 at 1:46
  • $\begingroup$ @gung I'm kicking myself for not linking to it first. $\endgroup$
    – Glen_b
    Nov 11, 2013 at 2:38
  • 1
    $\begingroup$ @Glen: Very good disposition. I had the same confusion for a long time thanks to the not so good treatment of the topic in textbooks and almost in a lot of resources on the internet. On the other hand unconditional distribution of Y is almost always examined to deduce a model for the conditional distribution, especially in time-series context. Is there a theoretical reasoning behind it? I tried to ask it as a question but I think couldn't phrase it properly: stats.stackexchange.com/questions/74886/… $\endgroup$ Nov 14, 2013 at 9:53
  • $\begingroup$ @CagdasOzgenc The only reason I can think of to do it is because it's easy to do before you have a model. The answer you have at that linked question looked like a good answer to me when it was posted. $\endgroup$
    – Glen_b
    Nov 14, 2013 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.