1
$\begingroup$

This question already has an answer here:

I understand that $y_{i}$s are normally distributed because we assume that the residual is normally distributed which seems a decent assumption.

Question: Does that mean $Y$ is also normally distributed? In other words, if I plot a histogram of $Y$ values will they have a unimodal Gaussian shape?

Confusion: We generally apply linear regression on problems that involve predicting a continuous variable. Given the wide applicability of linear regression, I am finding it little difficult to believe that $Y$ tend to be normally distributed for all of them.

I believe that in case of Logistic Regression, I can say that $Y$ will be Bernoulli distributed, and $y_{i}$s will have the same distribution. In other words, I can say that $Y$ has a Bernoulli distribution with $P(y_{i}=1) = \theta$

$\endgroup$

marked as duplicate by kjetil b halvorsen, mdewey, jbowman, AdamO, Michael Chernick Feb 9 '18 at 19:40

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    $\begingroup$ (1) You haven't made any assumptions about the regressor. What if it takes on only two values? This is common in experiments. You would then be asking whether the mixture distribution of the residuals is Normal. Obviously it won't be unless the slope is zero. (2) What exactly do you mean by the $y_i$ and $Y$? $\endgroup$ – whuber Feb 7 '18 at 23:19
  • $\begingroup$ By $y_{i}$, I mean an individual training instance. By $Y$ I mean all data in my training set. "y" and "Y" point to the dependent variable $\endgroup$ – The Wanderer Feb 7 '18 at 23:21
  • 1
    $\begingroup$ That's right: $Y$ can have a conditional Bernoulli distribution in logistic regression. Its parameter is allowed to depend on the regressors. $\endgroup$ – whuber Feb 8 '18 at 22:07
  • 2
    $\begingroup$ You can learn a great deal on our site. See stats.stackexchange.com/…. The standard book is Hosmer & Lemeshow, Applied Logistic Regression. It's also covered in many textbooks on regression modeling. $\endgroup$ – whuber Feb 8 '18 at 22:17
  • 2
    $\begingroup$ Y can have more or less any distribution, as long as the residuals of $\hat{Y}$ are normally distributed. You can see this by simulating non-normal $X$, and the simulating $Y_{i}= \beta_{0} + \beta_{X}X_{i} + \varepsilon_{i}$ where $\varepsilon$ is distributed normally. If the variance of $\varepsilon$ is small relative to the variance of $X$, (1) a histogram of $Y$ should reveal a non-normal distribution, and (2) regressing $Y$ on $X$ should provide $\hat{\beta_{0}}$ and $\hat{\beta_{X}}$ that are unbiased estimates of the original $\beta_{0}$ and $\beta_{X}$. $\endgroup$ – Alexis Feb 9 '18 at 19:49
4
$\begingroup$

$$y_i = a + \sum b_{j} x_{ij} + \epsilon_i$$

You can have a multitude of distribution types for $y_i$, $x_{ij}$ and $\epsilon_i$ in this linear regression. While the three, $y_i$, $x_{ij}$, and $\epsilon_i$, relate to each other, I guess that it is clear that when we only restrict/assume a single one of them, the $\epsilon_i \sim \mathcal{N(0,\sigma^2)}$ (which is the special case of linear regression called OLS, ordinary least squares, regression), that the $y_i$ and $x_i$ can still have some independent degree of variation between each other and do not need to be normal distributed like the error term $\epsilon$.


Example: Y can be modeled as a quadratic function of a normal distributed variable X plus some normal distributed error term. In this case you can clearly fit Y using OLS (fitting to a quadratic function) but Y is not normal distributed (since it is related to a sum of variables in which one of them is a quadratic function of a normal distribution.

an example image found in another stackexchange post: https://stats.stackexchange.com/a/30981/164061

example of quadratic plot


The variations/generalizations, of OLS, that you encounter in regression (such as GLM) assume

  • different (more general) distributions of the error term $\epsilon$
  • as well apply some non linear function of the ensemble of the linear part $\sum b_j x_{ij}$ (note this is different from the $x_{ij}$ which might be non-linear functions of some measured set of parameters such as in quadratic fits, these cases do not take non-linear manipulation of the entire linear sum)

The presence of these options might be confusing and generate the idea that OLS is for normal distributed $Y$ and GLM and those other cases for non-normal distributed $Y$. But, the case is more nuanced and OLS can also apply to non-normal distributed $Y$ (and more specifically is just about normal distributed error-term).

$\endgroup$
2
$\begingroup$

Regarding the statement marked question, for linear regression, if the vector of residuals $e \sim N(0, \sigma^2 I)$ then since $y = X \beta + e$ and $X \beta$ is non-random $y \sim N(X \beta, \sigma^2 I)$ so, yes, $y$ is normally distributed, as well.

Regarding the histogram, normally histograms are made from values that are independent and identically distributed but the observed $y_i$ values under the linear model have different distributions as their means differ so that does not fulfill the requirement for a histogram (except for the trivial linear model consisting of intercept only).

Regarding the statement marked confusion, you might want to look up the Gauss Markov theorem which provides an optimality condition for least squares without the assumption of normality.

Regarding logistic regression, that is not a special case of linear regression.

Logistic regression is an example of a generalized linear model (GLM) and GLMs form a family that is a superset which include both the linear model and logistic regression.

Linear regression (referred to in the subject of the post and above in this answer) refers to regression with a normally distributed response variable. The predictor variables and coefficients are fixed (i.e. non-random) and the residuals are normally distributed as well. In R one uses the lm function to analyze such models.

Generalized linear regression (GLM) is a superset of linear regression. The assumptions are somewhat similar to linear models but now the dependent variable belongs to an exponential family (not to be confused with the exponential distribution). That family includes the normal, binomial, exponential, poisson and other distributions. The mean of the response variable is related via a link function to a linear function of the independent variables (which is where the linearity comes in). If the dependent variable is normally distributed and the link function is the identity function then GLMs reduce to linear models. If the binomial distribution is assumed then the model is referred to as binomial logistic regression. This can encompass 0/1 data (binary logistic regression). In R one uses the glm function to analyze GLMs. Note that the general linear model is not the same as the generalized linear model (GLM) so be careful when reading accounts of this.

There are other directions in which one can generalize but normally these are not what one is referring to when one refers to linear regression and these have separate names. For example, one could assume that the $\beta$ values are random (random effects model) or that some are fixed and some are random (mixed effects models). Such models are fit by the lme4 package in R. These are closely related to Bayesian models. Other models include nonlinear regression models where the residuals are still normal and independent but $X \beta$ is replaced with a nonlinear function of $\beta$. The nls function in R fits such models. Measurement error models ($X$ is random), generalized mixed models (GLMs where coefficients are partly or entirely random), time series models (notably auto-correlation and ARIMA dependence structures among the residuals), multivariate response models ($Y$ is a matrix) and other models may have a form similar to linear regression models as well.

Regarding references, McCullagh and Nelder, Generalized Linear Models (second edition) is useful and has separate chapters for different types of data. For example, Chapter 4 is about binary data.

Another possibility is to find a book with a chapter on GLMs. Here are some at a variety of levels. Chapter 2 of N. Wood, Generalized Additive Models, is about GLMs and discusses both theory as well as examples using R code -- the book as a whole is about GAMs but this chapter is about GLMs. Zuur et al, Mixed Effect Models and Extensions in Ecology with R is a relatively non-mathematical book that discusses GLMs in Chapters 8, 9 and 10 together with R code. McCulloch et al, Generalized, Linear and Mixed Models provides a terse discussion of GLMs in Chapter 5.

Note

Note that a number of those commenting have claimed that $X$ is random. If you do make that assumption then of course the answer would change; however, as you were asking How is Y Normally Distributed in Linear Regression not only is that not the standard assumption but that route will lead to it not being distributed normal whereas the standard assumptions do lead to it being normal so they fulfill the statement of the question as summarized in its subject.

Another interpretation (not mine) is that you are asking about the distribution of the observed values $y_i$ as represented by their histogram. In that case as pointed out in comments under the question it has already been answered here What if residuals are normally distributed, but y is not? and the question is a duplicate. Again, be aware that that will not lead to the requested explanation of how Y is normal.

EDIT: Have expanded the discussion at the end and added a paragraph on the histogram.

Have moved comments to Note above as comments were getting too long and contentious.

$\endgroup$
  • 9
    $\begingroup$ Note that $Y$ is indeed normally distributed - but conditionally on $X$, not unconditionally, as OP states in the question. Unconditionally, $Y$ is almost never normally distributed, unless your predictor have no influence at all, which is typically not the case we are interested in. $\endgroup$ – Stephan Kolassa Feb 8 '18 at 7:52
  • 4
    $\begingroup$ Although the question does not explicitly mention $X$, in its depiction of "plotting a histogram of $Y$" it makes it very clear that the unconditional distribution is of concern, not the conditional one. In light of that, your answer looks rather misleading. $\endgroup$ – whuber Feb 8 '18 at 14:29
  • 3
    $\begingroup$ The "usual conditions" are what you state, but they do not imply $Y$ is normal. It seems one has to confuse $Y$ with its conditional distributions in order to draw the conclusions you make in this answer. $\endgroup$ – whuber Feb 8 '18 at 22:19
  • 1
    $\begingroup$ G.Grothendieck: $y$ is not in general Normally distributed in the sense that if you were to make a histogram of its observed values, that histogram could have any shape. More precisely, the marginal distribution of $y$, its distribution irrespective of the level of $X\beta$, is not (necessarily) Normal. Given a fixed value of $X\beta$, on the other hand, $y$ does follow a Normal distribution. $\endgroup$ – Ruben van Bergen Feb 9 '18 at 13:04
  • 2
    $\begingroup$ G.Grothendieck: You misunderstand me. I'm not referring to the difference between theory and practice. I'm joining Stephan Kolassa & whuber in pointing out the difference between the conditional & marginal distributions of $y$. $\endgroup$ – Ruben van Bergen Feb 9 '18 at 13:13
2
$\begingroup$

I will try to kill your confusion in the one dimensional case. The proof with $k$ regressors is simply an extension and does not affect the proof below.

We have ONLY the following hypotesis:

  1. $Y = \beta X + e $

  2. $e \sim N(0, \sigma^2)$

  3. $X$ and $e$ are independent

Given these hypotesis you can prove that $Y\mid X \sim N(\beta x, \sigma^2)$

BUT you CANNOT prove that $Y$ is Normal. In order to do that you need one additional assumption about the JOINT DISTRIBUTION of $X$ and $e$ that is $X$ and $e$ are JOINTLY Normal distributed. This of course implies that $X$ and $Y$ are JOINTLY Normal distributed as well.

We are not making any assumption about the distribution of $X$, and I want to point that $X$ is RANDOM unlike many people said that $X$ is NON RANDOM. That is not true and you'll do yourself no favor if you think that.

Consequently $X$ has a probability distribution. That being said let's begin with the proof:

$f_{Y \mid X}(y \mid x) = \dfrac{f_{YX}(y,x)}{f_{X}(x)}$

$f_{YX}(y,x)=f_{YX}(\beta X + e,x) = f_{eX}(y- \beta X,x)$

the last equality is true because it is a simple bivariate transformation

The independence of $e$ and $X$ implies that

$f_{eX}(y- \beta X,x) = f_e(y- \beta X)f_X(x)$

Finally, since $e \sim N(0, \sigma^2)$

$f_{Y \mid X}(y \mid x) = f_e(y- \beta X) = \dfrac{1}{2 \sigma \sqrt{(\pi)}}exp{\bigg(\dfrac{1}{2} \Big(\dfrac{y -\beta x}{\sigma} \Big)^2 \bigg)} $

This conclude the proof since the last equality implies that.

$ Y \mid X \sim N(\beta x, \sigma^2)$

To summarize: given the hypotesis in the OP, if you don't know the joint distribution of $X$ and $Y$ you CANNOT draw any conclusion about the UNCONDITIONAL of $Y$ BUT you can determine the CONDITIONAL DISTRIBUTION of $Y \mid X$ independently of the MARGINAL of $X$

$\endgroup$
  • $\begingroup$ Thank you Hard Core. How does that changes in case of logistic regression? $\endgroup$ – The Wanderer Feb 9 '18 at 16:03
  • $\begingroup$ The theory is the same. Of course in the logistic case you have a series of different hypotesis that make calculations more difficult. You have non normal distributed residual and non linear conditional expectation. But the principle to find the unconditional distribution of the response (Y) is exactly the same. You must know the joint and then find the marginal. Unfortunately in GLM you don't have hypotesys about the joint distribution. $\endgroup$ – Hard Core Feb 9 '18 at 18:07
0
$\begingroup$

Actually, your y scores need not be normally distributed. Even when y departs from normality, it is entirely possible for the residuals to assume normality, which is one of the assumptions of linear regression.

The distinction with logistic regression (or poisson or other), however, is a somewhat different issue. There you are specifying the mapping (or link) between changes in X and y scores.

$\endgroup$
  • $\begingroup$ Can you please elaborate a little further on the distinction with logistic regression? I think I am trying to draw the same parallel in linear regression as with logistic regression $\endgroup$ – The Wanderer Feb 7 '18 at 23:36
  • 1
    $\begingroup$ Is it sane to say that Y can belong to any distribution, not only Normal, and moreover it can be modeled really well with the use of OLS regression: given that X bares enough information about Y, and the residuals are normally distributed? $\endgroup$ – Alexey Burnakov Feb 9 '18 at 14:52
  • 1
    $\begingroup$ @AlexeyBurnakov Never said 'really well', just that the form of the parent need not be Gaussian for residuals to assume normality $\endgroup$ – HEITZ Feb 9 '18 at 15:32
  • $\begingroup$ @HEITZ, sure, you did not say. I see, thanks! $\endgroup$ – Alexey Burnakov Feb 9 '18 at 15:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.