1
$\begingroup$

Let $Y_1,Y_2,\cdots,Y_n$ be independently distributed random variables such that $Y_i\sim\mathcal N(\alpha+\beta x_i,\sigma^2)$ for all $i=1,\cdots,n$. If $\hat\alpha$ and $\hat\beta$ be the least square estimates of $\alpha$ and $\beta$ respectively, then what is the joint distribution of $(\hat\alpha,\hat\beta)$?

We consider the model $y=\alpha+\beta x+\epsilon$ where $y$ is stochastic and $x$ is non-stochastic.

We have the paired observations $(x_i,y_i)$ and we assume that the errors $\epsilon_i\stackrel{\text{i.i.d}}{\sim}\mathcal{N}(0,\sigma^2)$ for all $i$.

Define $s_{xx}=\sum (x_i-\bar x)^2,\,s_{yy}=\sum(y_i-\bar y)^2$ and $s_{xy}=\sum(x_i-\bar x)(y_i-\bar y)$.

From the normal equations we have $\hat\alpha=\bar y-\hat\beta\bar x$ and $\hat\beta=\dfrac{s_{xy}}{s_{xx}}$.

Transforming $\mathbf Y=(Y_1,\cdots,Y_n)\to(Z_1,\cdots,Z_n)=\mathbf Z$ such that $\mathbf Y=\mathbf{A\,Z}$ where $\mathbf A$ is an orthogonal matrix with its first two rows $\left(\frac{1}{\sqrt n},\cdots,\frac{1}{\sqrt n}\right)$ and $\left(\frac{x_1-\bar x}{\sqrt{s_{xx}}},\cdots,\frac{x_n-\bar x}{\sqrt{s_{xx}}}\right)$.

From the distribution of the $Z_i$'s, one can show that $\bar y\sim\mathcal N\left(\alpha+\beta\bar x,\frac{\sigma^2}{n}\right)$ and $\hat\beta\sim\mathcal N\left(\beta,\frac{\sigma^2}{s_{xx}}\right)$, both independently distributed of each other.

From this, one gets $\hat\alpha\sim\mathcal{N}\left(\alpha,\sigma^2\left(\frac{1}{n}+\frac{\bar x^2}{s_{xx}}\right)\right)$.

So we have the two least square estimates each having a univariate normal distribution. They are definitely not independent; I have found they have correlation $\text{Corr}(\hat\alpha,\hat\beta)=-\frac{\sqrt{n}\bar x}{\sqrt{\sum x_i^2}}$.

But how can I find the joint distribution of $(\hat\alpha,\hat\beta)$ from this? I cannot simply conclude that they are jointly normal. They are probably jointly normal but how does it follow?

And I would also like to know that happens to the joint distribution if $\mathbb E(Y_i)=\alpha+\beta(x_i-\bar x)$ instead. How should the above facts be modified?

The following related posts came up during a search:

Are estimates of regression coefficients uncorrelated?

Are the estimates of the intercept and slope in simple linear regression independent?

Sampling distribution of regression coefficients

But don't quite get the answer I am looking for.

Edit.

To clarify, I am looking for a derivation of the exact distribution of the least square estimates. The potential duplicate post linked here does not derive the exact distribution. It possibly has what the final result should be, which is why I linked it to this post in the first place.

I am not familiar with matrix notations used in these contexts, which seem to be in abundance in every answer here. If possible, a textbook reference for the desired result would be welcome.

$\endgroup$

marked as duplicate by Xi'an, Michael Chernick, kjetil b halvorsen, jbowman, Peter Flom regression May 23 '18 at 11:29

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ In a linear regression you get the covariance matrix of all parameters, did you look at it? $\endgroup$ – Aksakal May 22 '18 at 19:20
  • $\begingroup$ Here is the answer to your question. Do you understand it or need explanation? $\endgroup$ – Aksakal May 22 '18 at 19:24
  • $\begingroup$ @Aksakal I cannot even see where the answer is. This probably makes it difficult to understand that. A standard textbook reference would probably be more helpful. $\endgroup$ – StubbornAtom May 22 '18 at 19:31
  • 1
    $\begingroup$ I don't think that is a duplicate. This question seems to be actually asking how to derive the sampling distribution, especially, how to show that it's multivariate normal (if it is). The duplicate, despite the title, seems to be more about conceptual differences between the sampling distribution and Bayesian posterior distributions of the coefficients $\endgroup$ – Juho Kokkala May 23 '18 at 5:11
  • 1
    $\begingroup$ Since the estimates are explicitly a linear transformation of the responses, and it is well-known (and quoted extensively on this site) that linear transformations of multivariate Normal variables are Normal, the result follows from any multivariate normality assumption of the response. I'm sure this point has been made in many threads, although it would be hard to search for a specific answer given the ubiquity of references to normality and regression. $\endgroup$ – whuber Jun 13 '18 at 12:58
3
$\begingroup$

In matrix notation, the least square estimates are given by \begin{equation} \hat{\theta} = \theta + (X^\prime X)^{-1} X^\prime U \end{equation} where $U \sim N_n(0, \sigma^2 \mathbf{I_n})$ with $I_n$ is an identity matrix of size $n$ (number of observations).

For the sake of argument, suppose that $X$ is known, then is follows that \begin{equation} E[\hat{\theta}] = \theta + (X^\prime X)^{-1}X^\prime E[U] = \theta \end{equation} and \begin{equation} Var[\hat{\theta}] = (X^\prime X)^{-1}X^\prime Var[U] X(X^\prime X)^{-1} = \sigma^2 (X^\prime X)^{-1} \end{equation} Since $\hat{\theta}$ is a linear function of $U$, then it follows that \begin{equation} \hat{\theta} \mid X \sim N(\theta, \sigma^2 (X^\prime X)^{-1}) \end{equation}

In the univariate case, it follows that \begin{equation} \theta=\left[\begin{array}{c} \alpha\\ \beta \end{array}\right] \end{equation} and \begin{equation} X=\left[\begin{array}{cc} 1 & x_{1}\\ \vdots & \vdots\\ 1 & x_{n} \end{array}\right] \end{equation}

If you solve $(X^\prime X)^{-1}$ then you know the covariance matrix for $\hat{\theta}$ and, hence, the covariance between $\hat{\alpha}$ and $\hat{\beta}$, which is the off-diagonal element of $\sigma^2 (X^\prime X)^{-1}$.

$\endgroup$
0
$\begingroup$

The joint distribution of model parameters in ordinary least squares (OLS) regression is described here and here. The formula is:

$$(\hat\beta - \beta)\ \xrightarrow{d}\ \mathcal{N}\big(0,\;\sigma^2Q_{xx}^{-1}\big)$$ where $$Q_{xx} = X ^T X$$, and $X$ is the design matrix.

Note, that in this case we do not assume that errors are Gaussian. In a large sample CLT kicks in, and we don't need the normality to obtain the distribution.

In your case $\hat\alpha=\hat\beta_0$ and $\hat\beta=\hat\beta_1$, and the design matrix $X$ has two columns: ones (intercept) and variable $x$, i.e. row $i$ is $X_{i0}=1,X_{i1}=x_i$

$\endgroup$
  • $\begingroup$ I am rather looking for this bivariate normal distribution stated in this post:stats.stackexchange.com/questions/61235/…. Is this the same as what you have written? Don't know if this is the answer I should be looking for though. $\endgroup$ – StubbornAtom May 22 '18 at 19:46
  • $\begingroup$ It's the same thing $\endgroup$ – Aksakal May 22 '18 at 19:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.