Normality of dependent variable = normality of residuals?

Question

This issue seems to rear its ugly head all the time, and I'm trying to decapitate it for my own understanding of statistics (and sanity!).

The assumptions of general linear models (t-test, ANOVA, regression etc.) include the "assumption of normality", but I have found this is rarely described clearly.

I often come across statistics textbooks / manuals / etc. simply stating that the "assumption of normality" applies to each group (i.e., categorical X variables), and we should we examining departures from normality for each group.

Questions:

1. does the assumption refer to the values of Y or the residuals of Y?

2. for a particular group, is it possible to have a strongly non-normal distribution of Y values (e.g., skewed) BUT an approximately (or at least more normal) distribution of residuals of Y?

   Other sources describe that the assumption pertains to the residuals of the model (in cases where there are groups, e.g. t-tests / ANOVA), and we should be examining departures of normality of these residuals (i.e., only one Q-Q plot/test to run).

3. does normality of residuals for the model imply normality of residuals for the groups? In other words, should we just examine the model residuals (contrary to instructions in many texts)?

   To put this in a context, consider this hypothetical example:

   • I want to compare tree height (Y) between two populations (X).
   • In one population the distribution of Y is strongly right-skewed (i.e., most trees short, very few tall), while the other is virtually normal
   • Height is higher overall in the normally distributed population (suggesting there may be a 'real' difference).
   • Transformation of the data does not substantially improve the distribution of the first population.

4. Firstly, is it valid to compare the groups given the radically different height distributions?

5. How do I approach the "assumption of normality" here? Recall height in one population is not normally distributed. Do I examine residuals for both populations separately OR residuals for the model (t-test)?

---

Please refer to questions by number in replies, experience has shown me people get lost or sidetracked easily (especially me!). Keep in mind I am not a statistician; though I have a reasonably conceptual (i.e., not technical!) understanding of statistics.

P.S., I have searched the archives and read the following threads which have not cemented my understanding:

• ANOVA assumption normality/normal distribution of residuals
• Normality of residuals vs sample data; what about t-tests?
• Is normality testing 'essentially useless'?
• Testing normality
• Assessing normality of distribution
• What tests do I use to confirm that residuals are normally distributed?
• What to do when Kolmogorov-Smirnov test is significant for residuals of parametric test but skewness and kurtosis look normal?

Answer 1

One point that may help your understanding:

If $x$ is normally distributed and $a$ and $b$ are constants, then $y=\frac{x-a}{b}$ is also normally distributed (but with a possibly different mean and variance).

Since the residuals are just the y values minus the estimated mean (standardized residuals are also divided by an estimate of the standard error) then if the y values are normally distributed then the residuals are as well and the other way around. So when we talk about theory or assumptions it does not matter which we talk about because one implies the other.

So for the questions this leads to:

1. yes, both, either
2. No, (however the individual y-values will come from normals with different means which can make them look non-normal if grouped together)
3. Normality of residuals means normality of groups, however it can be good to examine residuals or y-values by groups in some cases (pooling may obscure non-normality that is obvious in a group) or looking all together in other cases (not enough observations per group to determine, but all together you can tell).
4. This depends on what you mean by compare, how big your sample size is, and your feelings on "Approximate". The normality assumption is only required for tests/intervals on the results, you can fit the model and describe the point estimates whether there is normality or not. The Central Limit Theorem says that if the sample size is large enough then the estimates will be approximately normal even if the residuals are not.
5. It depends on what question your are trying to answer and how "approximate" your are happy with.

Another point that is important to understand (but is often conflated in learning) is that there are 2 types of residuals here: The theoretical residuals which are the differences between the observed values and the true theoretical model, and the observed residuals which are the differences between the observed values and the estimates from the currently fitted model. We assume that the theoretical residuals are iid normal. The observed residuals are not i, i, or distributed normal (but do have a mean of 0). However, for practical purposes the observed residuals do estimate the theoretical residuals and are therefore still useful for diagnostics.

Answer 2

The short answers:

1. residuals
2. no
3. depends, both approaches have advantages and disadvantages
4. why not? It may make more sense to compare medians instead of means.
5. from what you have told us, the normality assumption is probably violated

The longer answer:

The assumption is that the dependent variable (y) is normally distributed but with different means for different groups. As a consequence, if you plot just the distribution of y it can easily look very different from your standard bell shaped normal curve. The residuals represent the distribution of y with those differences in means "filtered out".

Alternatively, you could look at the distribution of y in each group separately. This also filters out the differences in means across groups. The advantage is that this way you also get information about the distribution in each group, which in your case seems relevant. The disadvantage is that each group contains less observations than the combined dataset which you would get when looking at the residuals. Moreover, you would not be able to meaningfully compare groups if you have many groups, e.g. because you entered many predictor variables to your model or a (quasi-)continuous predictor variable to your model. So if your model consists of only one categorical predictor variable and the number of observations in each group is large enough, then it can be meaningful to inspect the distribution of y in each group separately. You can always inspect the residuals.

Answer 3

By definition of assumptions, the random variable $Y$ is a linear combination of $X$ and the residuals, all other things being constant.
If $X$ is not stochastic, and the error terms are normal, then $Y$ is normal and so are the residuals.

Question 1)
The assumptions refers to two things. First, to the normality of the error terms. Second, to the linearity and completeness of the model. Both things are necessary for inference. But if these assumptions are met, then both the residuals $e$ and $Y$ are normally distributed and the solution can be calculated quite easily, because they depend on the error terms $\epsilon$, given $X$.
For example the distribution of $Y$ in a regular OLS model might be $Y|X-N(X\beta,\sigma^2)$.
If your $X$ group is not normal, then this will potentially skew the unconditional $Y$. In fact this is very likely to happen. However, the important thing is that the distribution of $Y|X$ is normal.

Questions 2)
Yes it is possible to have skewed values for $Y$ because of the $X$. Yet, the residuals will be normal if all assumptions are met (how else could you do intervals and hypothesis testing?!). For this part of your question there is a pretty definitive answer in this thread: What if residuals are normally distributed, but y is not?

Question 3)
The important thing for using linear models requiring normality is that residuals which are not normal, wgether this is in a group or not, are an important indicator that your model might not fit your data.
If you are doing ANOVA, then of course your overall residuals don't have to be normal (or rather homoscedastic), that wouldn't make sense. In a regression though, you better have a model with ends up with overall normal residuals. If not, your interval estimators and tests will be wrong. This may be the case of certain autocorrelations, or a missing variable bias. If the model is 100% correct (including maybe structural breaks and weighting if necessary), it is not far fetched to assume normal error terms, even centered around 0. Practically the question often becomes: Can we get away with these things if the sample is large enough? There is no definitive answer, but for a 100% correct approach yes, all residuals should be normal.

Question 4 & 5)
It depends on what you mean by comparing. Given the assumption of normal error terms, you can test based on the the assumption of two different distributions. You can also use GLS estimation for a regression to account for the different distribution parameters - IF you have the right model... and I guess your groups themselves work as an indicator/binary variable?
Then it would probably be very hard to reason that the distribution of residuals will be normal - consequence is that while you can do stuff with your data, it will not be on the basis of regular OLS.
But it depends on what you want to do with the data.

The important thing is though: You still do not get to go around the assumptions of the linear model you are using. You can make issues better by assuming asymptotic large sample properties, but if I guess since you are asking for a definitive answer that is not what you have in mind.
In the case of your example, if you have data which might explain the skewedness you will regain normality in your residuals and in $Y|X$. But if you are just using binary indicators for a regression, you are essentially using the wrong model. You can indeed do tests with this, but when it comes to regression your interval results will be invalid, essentially you are missing data for a complete model.

I think a good approach would be to look into the algebra of regular OLS with a focus on the resulting distributions.

Answer 4

A clarification on Question 3: Normality of the residuals definitely does not imply normality within groups. The marginal distribution of residuals can be normal while the conditionals are not. This is true because a mixture of non-normal distributions can possibly be normal; see https://stats.stackexchange.com/a/486951/102879 for an example.

Answer 5

The assumptions refers to two things. First, to the normality of the error terms. Second, to the linearity and completeness of the model. Both things are necessary for inference. But if these assumptions are met, then both the residuals e and Y are normally distributed and the solution can be calculated quite easily, because they depend on the error terms ϵ, given X. For example the distribution of Y in a regular OLS model might be Y|X−N(Xβ,σ2). If your X group is not normal, then this will potentially skew the unconditional Y. In fact this is very likely to happen. However, the important thing is that the distribution of Y|X is normal.

is there any reference available for this?