1
$\begingroup$

I came across the assumptions of linear regression that said: -->The residuals should be normally distributed.

GLM(Generalized Linear model) assumes that target variable should follow one of the exponential family.

So does linear regression needs residuals as well as target variable to be distributed normally?

EDIT

https://online.stat.psu.edu/stat504/node/216/

In the above mentioned, it is written -

There are three components to any GLM:

  1. Random Component – refers to the probability distribution of the response variable (Y); e.g. normal distribution for Y in the linear regression, or binomial distribution for Y in the binary logistic regression.

Moreover in the assumption section,

The dependent variable Yi does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal,...)

I am new to machine learning, forgive me if i'm asking stupid question.

$\endgroup$
  • $\begingroup$ Answered several times on site already. No statistical procedure assumes the raw (unconditional) response variable has any specific distribution either for regression or for a GLM. I.e. looking at histograms, QQ plots or tests on the response itself is pointless. The assumption -for tests, CI and prediction intervals relates only to conditional distributions. $\endgroup$ – Glen_b Aug 26 at 4:52
  • $\begingroup$ I have edited my post, please have a look. The link that i have mentioned mentions that for linear regression, target variable should be normally distributed.(Also mentioned in question). $\endgroup$ – AYIKKATHIL KARTHIK Aug 27 at 10:02
  • $\begingroup$ Yes, I can point to more things like that, but those quotes are misleading you. Note the presence of $x_i$ (the vector of predictor values for obs. $i$, in the notation of your link) in the expression for the linear predictor (and hence the mean). This means that $y_i|{x}_i$ typically has a different distribution-parameter from $y_j|{x}_j$ (unless their x-vectors are identical), and combining them all (by looking at the marginal distribution) you end up with a mixture of different distributions (same family but different parameters), which is not the same as the conditional distribution... $\endgroup$ – Glen_b Aug 28 at 2:01
  • $\begingroup$ ... while $y_i|{x}_i$ may be Gaussian (or Gamma or Poisson or binomial etc depending on the GLM), the collection of random variables across different $x$-vectors is not Gaussian (or Gamma or Poisson etc); it depends on the distribution of the $x$'s. Again, I encourage you to try some searches, since this is a pretty well-worn topic on this site. $\endgroup$ – Glen_b Aug 28 at 2:05
1
$\begingroup$

It depends on what you’re doing. If you just want to predict, then it doesn’t matter. The Gauss-Markov theorem does not say anything about a normal error term.

However, when the error term is normal, then the OLS estimator $\hat{\beta}$ is the maximum likelihood estimator. If you don’t know about MLEs, you’ll see them over and over as you dive into statistics, but maximum likelihood is a nice property for many reasons.

Among those reasons is that the inferential methods like p-values on coefficients and F-tests of nested models come into play.

So if you want to do some kind of ANOVA, for example, the normality of the error term matters because you’re doing inference, not prediction.

The pooled distribution of the response variable (all of your $y$s) definitely does not have to be normal, even to get that maximum likelihood property and do inference, and the predictor variables definitely don’t have to be normal. Predictors often cannot be normal, such as when they are categorical variables e.g. male/female, treatment/control, etc.

EDIT

We often talk about normal residuals. This is casual language, and experienced statisticians know what is meant, but the residuals are a discrete distribution and cannot be normal. What we assume is a normal error term, and we use the residuals to gauge if that is a good assumption or not.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The only normality assumption in linear regression if you intend to do any testing is that the residuals be normally distributed. In simple linear regression with only one variable in the model, this implies that the independent variable must also be normally distributed. In multiple regression, however, the residuals must be normally distributed.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This is incorrect. The Gauss-Markov theorem does not require normality. $\endgroup$ – Dave Aug 26 at 3:41
  • $\begingroup$ The question wasn't about Gauss-Markov theorem. The question is about regression and all that it entails. $\endgroup$ – StatsStudent Aug 26 at 3:43
  • $\begingroup$ Gauss-Markov is about regression. $\endgroup$ – Dave Aug 26 at 3:44
  • $\begingroup$ I'm well aware of what Gauss-Markov is. It's about a certain aspect of regression, but not all it and I'll take the OP's question to broadly mean all (or even typical) aspects of ordinary regression. $\endgroup$ – StatsStudent Aug 26 at 3:45
  • $\begingroup$ So how is it that we get the BLUE from Gauss-Markov, whether the residuals are normal or not, yet normal residuals are required for linear regression? $\endgroup$ – Dave Aug 26 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.