In linear regression: Why does the sampling distribution of OLS estimators depend on the underlying distribution of errors? Thanks!

(I am relatively new to statistics in general, so more simply worded answers would be appreciated :)!)


Since you want simple answers, let's simplify the situation to the simplest regression model -- an intercept only model (~1 in R).

Then under least squares the parameter estimate will be the sample mean.

The distribution of the sample mean depends on the distribution of the population the sample was drawn from.

The sample mean is just 1/n times the sum, and for independent continuous (/discrete) variates, the distribution of the sum is the convolution of the pds (/pmfs).

So for example (assuming i.i.d. throughout):

  • if the population distribution is exponential$(1/\mu)$ (rate parameterization), the distribution of the sum is gamma$(n,1/\mu)$ (shape-rate parameterization)

  • if the population distribution is uniform, the sum of two is triangular and the sum of n of them is Irwin-Hall (or more generally, a location-scale family based on the Irwin-Hall)

  • if the population distribution is Poisson, the sum is Poisson

  • if the population distribution is Bernoulli, the sum is binomial

  • if the population distribution is normal, the sum is normal

and so on.

With simple or multiple regression the vector of parameters is a linear function of the data:

$$\hat{\beta} = Ay, \quad\text{where }\: A = (X^\top X)^{-1}X^\top$$

and as such will also be different for different conditional distributions for the response $f(Y|\underline{x})$, though we won't have convenient names for most of the distributions of weighted sums or weighted averages.

In sufficiently large samples you may be able to argue that the coefficient estimates will be approximately normal.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.