Where does the misconception that Y must be normally distributed come from?

Question

Seemingly reputable sources claim that the dependent variable must be normally distributed:

Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)$, and independent, and $X$ is fixed, and constant variance $\sigma^2$.

Penn State, STAT 504 Analysis of Discrete Data

Secondly, the linear regression analysis requires all variables to be multivariate normal.

StatisticsSolutions, Assumptions of Linear Regression

This is appropriate when the response variable has a normal distribution

Wikipedia, Generalized linear model

Is there a good explanation for how or why this misconception has spread? Is its origin known?

Related

• Linear regression and assumptions about response variable

Answer 1

'Y must be normally distributed'

must?

---

In the cases that you mention it is sloppy language (abbreviating 'the error in Y must be normally distributed'), but they don't really (strongly) say that the response must be normally distributed, or at least it does not seem to me that their words were intended like that.

The Penn State course material

speaks about "a continuous variable $Y$", but also about "$Y_i$" as in $$E(Y_i) = \beta_0 + \beta_1 x_i$$ where we could regard $Y_i$, which is as amoeba called in the comments 'conditional', normally distributed,

$$Y_i \sim N(\beta_0 + \beta_1x_i,\sigma^2)$$

The article uses $Y$ and $Y_i$ interchangeably. Throughout the entire article one speaks about the 'distribution of Y', for instance:

• when explaining some variant of GLM (binary logistic regression),

  Random component: The distribution of $Y$ is assumed to be $Binomial(n,\pi)$,...

• in some definition

  Random Component – refers to the probability distribution of the response variable ($Y$); e.g. normal distribution for $Y$ in the linear regression, or binomial distribution for $Y$ in the binary logistic regression.

however at some other point they also refer to $Y_i$ instead of $Y$:

• The dependent variable $Y_i$ does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal,...)

The statisticssolutions webpage

is an extremely brief, simplified, stylized description. I am not sure you should take this serious. For instance, it speaks about

..requires all variables to be multivariate normal...

so that is not just the response variable,

and also the the 'multivariate' descriptor is vague. I am not sure how to get that interpreted.

The wikipedia article

has an additional context explained in brackets:

Ordinary linear regression predicts the expected value of a given unknown quantity (the response variable, a random variable) as a linear combination of a set of observed values (predictors). This implies that a constant change in a predictor leads to a constant change in the response variable (i.e. a linear-response model). This is appropriate when the response variable has a normal distribution (intuitively, when a response variable can vary essentially indefinitely in either direction with no fixed "zero value", or more generally for any quantity that only varies by a relatively small amount, e.g. human heights).

This 'no fixed zero value' seems to point to the case that a linear combination $y+\epsilon$ when $\epsilon \sim N(0,\sigma)$ has an infinite domain (from minus infinity to plus infinity) whereas often many variables have some finite cut-off value (such as counts not allowing negative values).

The particular line has been added on March 8 2012, but note that the first line of the Wikipedia article still reads "a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution" and is not so much (not everywhere) wrong.

---

Conclusion

So, based on these three examples (which indeed could generate misconceptions, or at least could be misunderstood) I would not say that "this misconception has spread". Or at least it does not seem to me that the intention of those three examples is to argue that Y must be normally distributed (although I do remember this issue has arised before here on stackexchange, the swap between normally distributed errors and normally distributed response variable is easy to make).

So, the assumption that 'Y must be normally distributed' seems to me not like a widespread believe/misconception (as in something that spreads like a red herring), but more like a common error (which is not spread but made independently each time).

---

Additional comment

An example of the mistake on this website is in the following question

What if residuals are normally distributed, but y is not?

I would consider this as a beginners question. It is not present in the materials like the Penn State course material, the Wikipedia website, and recently noted in the comments the book 'Extending the Linear Regression with R'.

The writers of those works do correctly understand the material. Indeed, they use phrases such as 'Y must be normally distributed', but based on the context and the used formulas you can see that they all mean 'Y, conditional on X, must be normally distributed' and not 'the marginal Y must be normally distributed'. They are not misconceiving the idea themselves, and at least the idea is not widespread among statisticians and people that write books and other course materials. But misreading their ambiguous words may indeed cause the misconception.

Answer 2

Is there a good explanation for how/why this misconception has spread? Is its origin known?

We generally teach undergraduates a "simplified" version of statistics in many disciplines. I am in psychology, and when I try to tell undergraduates that p-values are "the probability of the data—or more extreme data—given that the null hypothesis is true," colleagues tell me that I am covering more detail than I need to cover. That I am making it more difficult than it has to be, etc. Since students in classes have such a wide range of comfort (or lack thereof) with statistics, instructors generally keep it simple: "We consider it to be a reliable finding if p < .05," for example, instead of giving them the actual definition of a p-value.

I think this is where the explanation for why the misconception has spread. For instance, you can write the model as:

$Y = \beta_0 + \beta_1X + \epsilon$ where $\epsilon \sim \text{N}(0, \sigma^2_\epsilon)$

This can be re-written as:

$Y|X \sim \text{N}(\beta_0 + \beta_1X, \sigma^2_\epsilon)$

Which means that "Y, conditional on X, is normally distributed with a mean of the predicted values and some variance."

This is difficult to explain, so as shorthand people might just say: "Y must be normally distributed." Or when it was explained to them originally, people misunderstood the conditional part—since it is, honestly, confusing.

So in an effort to not make things terribly complicated, instructors just simplify what they are saying as to not overly confuse most students. And then people continue on in their statistical education or statistical practice with that misconception. I myself didn't fully understand the concept until I started doing Bayesian modeling in Stan, which requires you to write your assumptions in this way:

model {
  vector[n_obs] yhat;

  for(i in 1:n_obs) {
    yhat[i] = beta[1] + beta[2] * x1[i] + beta[3] * x2[i];
  }

  y ~ normal(yhat, sigma);
}

Also, in a lot of statistical packages with a GUI (looking at you, SPSS), it is easier to check if the marginal distribution is normally distributed (simple histogram) than it is to check if the residuals are normally distributed (run regression, save residuals, run histogram on those residuals).

Thus, I think the misconception is mainly due to instructors trying to shave off details to keep students from getting confused, genuine—and understandable—confusion among people learning it the correct way, and both of these reinforced by ease of checking marginal normality in the most user-friendly statistical packages.

Answer 3

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.

---

Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.