2
$\begingroup$

In the context of the classical linear regression model (with all standard assumptions), we know that when the error term is normally distributed, least squares is minimum variance among all linear unbiased estimators and we know the exact statistical distribution of the t statistics.

However, if the residual histogram suggests that the error term is something other than normal, then the t statistics will not have the claimed t distribution (nor will the F statistic be F distributed). Surely, it does not make sense to assume normality if the residuals suggest otherwise. Instead, if we want to know the theoretical distribution of the t statistic, why not assume that the errors are distributed in population based on what we observe from the residual histogram?

Improve this question
$\endgroup$
12
  • $\begingroup$ The short answer to that is that testing the significance of the parameter estimate(s) implicitly assumes that the residuals are normally distributed, because that's how the test statistic was determined. Sure, you can test the residuals to see if the normal error assumption is reasonable, but that's not the same thing--that's assumption checking, not model selection. $\endgroup$
    – heropup
    Commented Jan 22, 2014 at 1:46
  • $\begingroup$ You can bootstrap from the residuals to get a significance value. $\endgroup$
    – Jeremy Miles
    Commented Jan 22, 2014 at 2:16
  • 2
    $\begingroup$ What hypothesis are you testing there? $\endgroup$
    – Glen_b
    Commented Jan 22, 2014 at 2:22
  • $\begingroup$ @ Glen_b - This is a theoretical question $\endgroup$
    – Christian
    Commented Jan 22, 2014 at 3:01
  • $\begingroup$ @ heropu - Yes, generally we assume that the errors are normally distributed since we need this assumption for exact inference. However, what are the implications if we construct signficance tests based on a bootstrap distribution using the observed distribution of residuals? $\endgroup$
    – Christian
    Commented Jan 22, 2014 at 3:03

1 Answer 1

Reset to default
3
$\begingroup$

The Central Limit Theorem applies in this case. If the residuals are not normally distributed, but the sample size is large enough, then the t statistics will be approximately t-distributed (and the F statistic will be approximately F distributed). How good the approximation is depends on how different the residuals are from the normal and how large the sample size is. Many regression problems have a combination that makes the approximation reasonable.

If there is a reason to believe a different distribution, then there are methods to fit regression models using that assumption. GLM models can fit binomial, poisson, and gamma distributed y's and using maximum likelihood or Bayesian methods (or others) can allow you to fit other distributions.

But if you are unwilling to assume normality, how can you be sure of other distributions? Sometimes it is clear, but if the residuals look like it might be a gamma, but you are not sure, then fitting based on a normal may be just as good (because of the CLT) as fitting to a gamma that does not actually fit.

If you don't want to make assumptions about the distribution of the residuals then there are options like permutation tests or bootstrapping (or other non-parametric regression tools), but all of these have their own sets of assumptions and conditions where they may work better or worse.

In the end it is important what question you are trying to answer and what you know about the science that produced the data that are the most important.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.