I know that linear regression is based on the assumption that the errors are normally distributed (from both bayesian and classical views). I'm just trying to verify this assumption based on the final model.

Assume I've got 3 normal random variables x1, x2, x3. I can regress (linearly) x1 on x2, x3 and get a linear regression model of the form:

x1 = b0 + b1*x2 + b2*x3 + e.

Reorganizing, e = x1 - b0 - b1*x2 - b2*x3.

Here, if I estimate bi's using least squares method, I can say 'e' is a linear combination of normally distributed variables, so it's normally distributed.

But if I estimate bi's using a Bayesian method and assume bi's also follow normal distribution, then 'e' is no longer a linear combination of normal random variables. Effectively, it's sum of normally (b0) and product-normally (b1*x2, b2*x3) distributed variables, and product-normal distribution is not normal in general. (http://math.stackexchange.com/questions/101062/is-the-product-of-two-gaussian-random-variables-also-a-gaussian)

Is there anything improper in above reasoning? How else can I try to validate the normal assumption for the error term?

  • $\begingroup$ Bayesians may make the same assumptions about the error terms in linear regression but the inference depends on priors on the regression parameters. Frequentists don't use priors. $\endgroup$ – Michael R. Chernick Oct 9 '12 at 17:19

When you do the Bayesian analysis are you allowing x1, x2, and x3 to vary/be updated? or are they fixed throughout the analysis?

Most analyses that I have seen, Frequentist or Bayesian, will treat the observed data as fixed and compute the estimates of the coefficients conditional on the observed data. So in that case you can think about the x's as being fixed and not worry about products of normals.

You should also note that while regression models (both Frequentist and Bayesian) assume that the "True" errors are iid Normal, or than the respones is independent normal with mean based on the regression equation, neither assume that the observed residuals are iid Normal (else why would there be standardized, studentized, etc. residuals?). Focusing too much on the distributional properties of the observed residuals may lead away from enlightenment rather than too it.

| cite | improve this answer | |
  • $\begingroup$ I read the 1st line of your response and understood where I was going wrong! Thanks :) My x1, x2, x3 are fixed throughout the analysis implying they are not to be treated as random for Bayesian analysis. I'm not sure about treating x's as fixed in Frequentist view. isn't regression of y on x's same as finding conditional expectation of y i.e. E[y|x1,x2,x3] ? (I'm still a beginner in stats and not very clear about these ideas, please correct me if I'm wrong..) $\endgroup$ – steadyfish Oct 11 '12 at 20:22
  • 1
    $\begingroup$ @steadyfish, You can derive the regression equations by either assuming the x's are fixed values and the y's are random based on the regression equation, or you can assume that x and y follow a multivariate normal distribution. Either way leads to the same set of regression equations so we generally don't distinguish between the cases and just treat the x's as fixed. Some purists may see a philosophical difference between the 2 approaches however (and which bootstrap method to use could depend on which way you think of the data). $\endgroup$ – Greg Snow Oct 12 '12 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.