45
$\begingroup$

Seemingly reputable sources claim that the dependent variable must be normally distributed:

Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)$, and independent, and $X$ is fixed, and constant variance $\sigma^2$.

Penn State, STAT 504 Analysis of Discrete Data

Secondly, the linear regression analysis requires all variables to be multivariate normal.

StatisticsSolutions, Assumptions of Linear Regression

This is appropriate when the response variable has a normal distribution

Wikipedia, Generalized linear model

Is there a good explanation for how or why this misconception has spread? Is its origin known?

Related

$\endgroup$
  • 17
    $\begingroup$ Sad. You're doing a good deed here... $\endgroup$ – jbowman Apr 25 '18 at 20:17
  • 7
    $\begingroup$ I don't know of any situation using linear regression that requires the marginal distribution of $Y$, or the joint of all variables be multivariate normal. Those look like misconceptions to me. $\endgroup$ – Matthew Drury Apr 25 '18 at 20:24
  • 8
    $\begingroup$ @MichaelChernick "Y is is normally distributed" is patently false. Check it out in R: X <- runif(n=100) then Y <- 3 + .5*X + rnorm(n=100, mean = 0, sd = .1) then play with histograms to convince yourself that neither X nor Y are normally distributed. Then summary(lm(Y ~ X)), and pay very close attention to how close the intercept is to 3, and the slope of X is to 0.5. The assumption is that the errors are normally distributed. $\endgroup$ – Alexis Apr 25 '18 at 20:56
  • 9
    $\begingroup$ @Alexis I believe what Michael was trying to say is that the multivariate Normality assumptions are sufficient but not necessary. That's clearly how one ought to read the Wikipedia quotation. The second quotation obviously is wrong in asserting those assumptions are necessary. The first quotation is ambiguous but could generously be read in the sense elucidated by Michael. $\endgroup$ – whuber Apr 25 '18 at 21:09
  • 6
    $\begingroup$ All I was saying was that normality assumption implies certain properties. For example in simple linear regression if you assume the error terms are i.i.d. normal with zero mean and constant variance the least squares estimate of the regression parameters are maximum likelihood. Keeping all the assumptions except normality least squares is no longer maximum likelihood but is still minimum variance unbiased. $\endgroup$ – Michael Chernick Apr 25 '18 at 22:08
13
$\begingroup$

'Y must be normally distributed'

must?


In the cases that you mention it is sloppy language (abbreviating 'the error in Y must be normally distributed'), but they don't really (strongly) say that the response must be normally distributed, or at least it does not seem to me that their words were intended like that.

The Penn State course material

speaks about "a continuous variable $Y$", but also about "$Y_i$" as in $$E(Y_i) = \beta_0 + \beta_1 x_i$$ where we could regard $Y_i$, which is as amoeba called in the comments 'conditional', normally distributed,

$$Y_i \sim N(\beta_0 + \beta_1x_i,\sigma^2)$$

The article uses $Y$ and $Y_i$ interchangeably. Throughout the entire article one speaks about the 'distribution of Y', for instance:

  • when explaining some variant of GLM (binary logistic regression),

    Random component: The distribution of $Y$ is assumed to be $Binomial(n,\pi)$,...

  • in some definition

    Random Component – refers to the probability distribution of the response variable ($Y$); e.g. normal distribution for $Y$ in the linear regression, or binomial distribution for $Y$ in the binary logistic regression.

however at some other point they also refer to $Y_i$ instead of $Y$:

  • The dependent variable $Y_i$ does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal,...)

The statisticssolutions webpage

is an extremely brief, simplified, stylized description. I am not sure you should take this serious. For instance, it speaks about

..requires all variables to be multivariate normal...

so that is not just the response variable,

and also the the 'multivariate' descriptor is vague. I am not sure how to get that interpreted.

The wikipedia article

has an additional context explained in brackets:

Ordinary linear regression predicts the expected value of a given unknown quantity (the response variable, a random variable) as a linear combination of a set of observed values (predictors). This implies that a constant change in a predictor leads to a constant change in the response variable (i.e. a linear-response model). This is appropriate when the response variable has a normal distribution (intuitively, when a response variable can vary essentially indefinitely in either direction with no fixed "zero value", or more generally for any quantity that only varies by a relatively small amount, e.g. human heights).

This 'no fixed zero value' seems to point to the case that a linear combination $y+\epsilon$ when $\epsilon \sim N(0,\sigma)$ has an infinite domain (from minus infinity to plus infinity) whereas often many variables have some finite cut-off value (such as counts not allowing negative values).

The particular line has been added on March 8 2012, but note that the first line of the Wikipedia article still reads "a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution" and is not so much (not everywhere) wrong.


Conclusion

So, based on these three examples (which indeed could generate misconceptions, or at least could be misunderstood) I would not say that "this misconception has spread". Or at least it does not seem to me that the intention of those three examples is to argue that Y must be normally distributed (although I do remember this issue has arised before here on stackexchange, the swap between normally distributed errors and normally distributed response variable is easy to make).

So, the assumption that 'Y must be normally distributed' seems to me not like a widespread believe/misconception (as in something that spreads like a red herring), but more like a common error (which is not spread but made independently each time).


Additional comment

An example of the mistake on this website is in the following question

What if residuals are normally distributed, but y is not?

I would consider this as a beginners question. It is not present in the materials like the Penn State course material, the Wikipedia website, and recently noted in the comments the book 'Extending the Linear Regression with R'.

The writers of those works do correctly understand the material. Indeed, they use phrases such as 'Y must be normally distributed', but based on the context and the used formulas you can see that they all mean 'Y, conditional on X, must be normally distributed' and not 'the marginal Y must be normally distributed'. They are not misconceiving the idea themselves, and at least the idea is not widespread among statisticians and people that write books and other course materials. But misreading their ambiguous words may indeed cause the misconception.

$\endgroup$
  • 3
    $\begingroup$ +1 That said: I think we've all seen lots of questions asserting the marginal normality of Y around here... there is some spread of misconception. :) $\endgroup$ – Alexis Apr 26 '18 at 2:10
  • $\begingroup$ Yes I agree that the assumption of 'y normally distributed' occurs often (I could not find examples easily, but that might be because people describe these things in between the lines and not with simple keywords). However, I believe that this is more something that is 'common' not something that is so much 'being spread out'. And at least, certainly the three examples given by the OP are not very strong (not strong in the sense of indicating the spreading of the misconception, although they do describe pathological use of language and how the errors may originate). $\endgroup$ – Martijn Weterings Apr 26 '18 at 2:25
  • $\begingroup$ @Martijn Weterings: I would like to disagree with your statement "I would not say that this misconception has spread". In his book Extending the Linear Regression with R, used as required reading in a number of graduate statistics programs, Julian Faraway states on page xi in the Preface of this book that "The standard linear model cannot handle non-normal responses, y, such as counts or proportions". $\endgroup$ – ColorStatistics Feb 6 at 15:57
  • $\begingroup$ @ColorStatistics, note the context and interpretation that I give to 'widespread' (as in something that spreads like a red herring). People make mistakes, and these mistakes might be ubiquitous. But it is not like spreading out as in getting copied (e.g. an example of a mistake that got copied, and spread out was the use of $n-1$ degrees of freedom in contingency tables instead of $(r-1)(c-1)$, which occurred between 1900 and 1920)..... $\endgroup$ – Martijn Weterings Feb 6 at 16:04
  • 1
    $\begingroup$ @ColorStatistics, I just went through some parts of the text and it is clear that the author is not confused (based on the formulae which are not ambiguous). E.g. the book even starts with: "$y = \beta_0 + \beta_1 x_1 + ... \beta_p x_p + \epsilon$ where $\epsilon$ is normally distributed". Indeed, the writer often uses such phrases as "the response is ... distributed". But, meaning the conditional response. I consider this more as shorthanded writing and the writer does not mean to convey literally that the marginal response should have the particular distribution that is mentioned. $\endgroup$ – Martijn Weterings Feb 6 at 16:27
29
$\begingroup$

Is there a good explanation for how/why this misconception has spread? Is its origin known?

We generally teach undergraduates a "simplified" version of statistics in many disciplines. I am in psychology, and when I try to tell undergraduates that p-values are "the probability of the data—or more extreme data—given that the null hypothesis is true," colleagues tell me that I am covering more detail than I need to cover. That I am making it more difficult than it has to be, etc. Since students in classes have such a wide range of comfort (or lack thereof) with statistics, instructors generally keep it simple: "We consider it to be a reliable finding if p < .05," for example, instead of giving them the actual definition of a p-value.

I think this is where the explanation for why the misconception has spread. For instance, you can write the model as:

$Y = \beta_0 + \beta_1X + \epsilon$ where $\epsilon \sim \text{N}(0, \sigma^2_\epsilon)$

This can be re-written as:

$Y|X \sim \text{N}(\beta_0 + \beta_1X, \sigma^2_\epsilon)$

Which means that "Y, conditional on X, is normally distributed with a mean of the predicted values and some variance."

This is difficult to explain, so as shorthand people might just say: "Y must be normally distributed." Or when it was explained to them originally, people misunderstood the conditional part—since it is, honestly, confusing.

So in an effort to not make things terribly complicated, instructors just simplify what they are saying as to not overly confuse most students. And then people continue on in their statistical education or statistical practice with that misconception. I myself didn't fully understand the concept until I started doing Bayesian modeling in Stan, which requires you to write your assumptions in this way:

model {
  vector[n_obs] yhat;

  for(i in 1:n_obs) {
    yhat[i] = beta[1] + beta[2] * x1[i] + beta[3] * x2[i];
  }

  y ~ normal(yhat, sigma);
}

Also, in a lot of statistical packages with a GUI (looking at you, SPSS), it is easier to check if the marginal distribution is normally distributed (simple histogram) than it is to check if the residuals are normally distributed (run regression, save residuals, run histogram on those residuals).

Thus, I think the misconception is mainly due to instructors trying to shave off details to keep students from getting confused, genuine—and understandable—confusion among people learning it the correct way, and both of these reinforced by ease of checking marginal normality in the most user-friendly statistical packages.

$\endgroup$
  • 2
    $\begingroup$ I think you're correct. Many people don't understand the conditional part. They just think normal distributed. $\endgroup$ – SmallChess Apr 26 '18 at 2:32
  • 3
    $\begingroup$ I agree that this might be 'one' of the modes by which this error occurs/spreads. The Penn State course material however seems to me as not due to this 'intentional' simplification and is also due sloppy notation writing. It is a bit like tiny (course) notes. Or like comments to stackexchange, simplifications in language. In some places they do use the correct words. (personally, my schematics/diagrams are better than my words/formulae, but that does not mean that what I write, if it is wrong, is necessarily a wrong idea) $\endgroup$ – Martijn Weterings Apr 26 '18 at 2:33
  • 1
    $\begingroup$ @MartijnWeterings Agreed—it is very easy to confuse someone by not using specific language. It is difficult to always be specific with your language in something as abstract as statistical assumptions, and many smart people make simple mistakes, leading to widespread misconceptions like this. $\endgroup$ – Mark White Apr 26 '18 at 2:36
  • 1
    $\begingroup$ MarkWhite, I really appreciate the attention you direct to how we teach... I think that speaks in an important way to the OP's interest in "spread of misconception" (in addition to the nuances of what is and what isn't a misconception). $\endgroup$ – Alexis Apr 26 '18 at 15:17
16
$\begingroup$

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.


Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.

$\endgroup$
  • 1
    $\begingroup$ I found it very insightful the way you introduced stronger assumptions one by one and described the implications. $\endgroup$ – ColorStatistics Feb 6 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.