While reading a textbook on regression I encountered the following paragraph:

The least squares estimate of a vector of linear regression coefficients ($\beta$) is

$$ \hat{\beta} = (X^{t}X)^{-1}{X^t}y $$

which, when viewed as a function of data $y$ (considering the predictors $X$ as constants), is a linear combination of the data. Using the Central Limit Theorem, it can be shown that the distribution of $\beta$ will be approximately multivariate normal if the sample size is large.

I'm definitely missing something from the text, but I don't understand how can a single $\beta$ value have a distribution? How are the multiple $\beta$ values generated to obtain the distribution referred to in the text?

  • 4
    $\begingroup$ $\beta$ is the vector of regression coefficients - does that clear up the confusion? $\endgroup$ – Macro May 17 '12 at 17:43
  • 5
    $\begingroup$ When using the least squares approach, you assume that $\beta$ is fixed but unknown. However, $\hat{\beta}$, since it's a function of the (random) data, has a distribution. Asymptotically the distribution is a normal distribution. Non-asymptotically, an individual coefficient will be a t distribution. $\endgroup$ – Taylor May 17 '12 at 17:49
  • 7
    $\begingroup$ It may help to observe that $H = (X^tX)^{-1}X^t$ is considered a constant matrix in the regression setting and that $y$ is the realization of a (vector valued) random variable. That bit about the CLT, though, is not quite correct: it is relying either on $H$ having a certain structure, which sometimes does not actually occur even with huge datasets, or else on $y$ itself being multivariate normal (but then there's no need to invoke the CLT). $\endgroup$ – whuber May 17 '12 at 18:08
  • 2
    $\begingroup$ @Taylor The individual component of the beta vactor will have a t distribution only if the error compoent in the regression model is Gaussian with 0 mean and constant variance. In the non-normal case you wouldn't necessarily known its distribution under the null hypothesis but it may still be asymptotically normal. However as whuber states the central limit theorem may not hold because it is a weighted average and we need to know that the weights don't chnage with sample size in a way that allows a few terms to dominate the sum. $\endgroup$ – Michael R. Chernick May 17 '12 at 22:18
  • 1
    $\begingroup$ @Wet You appear to be responding to an incorrect comment. When $Y-X\beta$ has a Normal distribution (the usual assumption), then the formula for $\hat\beta$ gives it as an explicit linear combination of $Y.$ Thus, in either the fixed or random regressor formulations, $\hat\beta$ has a (multivariate) Normal distribution. Student's t doesn't enter the picture until you start dividing estimates by their standard errors to test hypotheses. $\endgroup$ – whuber Feb 18 at 14:03

Not $\beta$ has a distribution but $\hat\beta$, as indicated by Taylor. The distribution of $\hat\beta$ stems from the fact that you get different $\hat\beta$ for different samples.---You can estimate this distribution based on the single $\hat\beta$ received from your single sample on condition that you have some information concerning the distribution of the underlying data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.