0
$\begingroup$

If $$Y_i\sim\mathrm{n}(\beta_0+\beta_1x_i,\sigma^2),\quad i=1,\cdots,n,$$ $$Y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad i=1,\cdots,n,$$

The maximum likelihood estimates of the three parameters:

$$\hat\beta_1=\frac{S_{xy}}{S_{xx}}$$

$$\hat\beta_0=\bar{y}-\hat\beta_1\bar{x}$$

$$\hat\sigma^2=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat\beta_0-\hat\beta_1x_i)^2$$

I tried to calculate the expectation of MLE variance of this linear model: $$\begin{align} \mathbb E \hat\epsilon_i^2 &=\mathbb E(Y_i-\hat\beta_0-\hat\beta_1x_i)^2\\ &=\mathbb E\left[(Y_i-\beta_0-\beta_1x_i)-(\hat\beta_0-\beta_0)-x_i(\hat\beta_1-\beta_1)\right]^2\\ &=\mathrm{Var}Y_i+\mathrm{Var}\hat\beta_0+x_i^2\mathrm{Var}\hat\beta_1-2\mathrm{Cov}(Y_i,\hat\beta_0)-2x_i\mathrm{Cov}(Y_i,\hat\beta_1)+2x_i\mathrm{Cov}(\hat\beta_0,\hat\beta_1)\\ \end{align}$$

$$\mathbb E\hat\sigma^2=\frac{1}{n}\sum_{i=1}^{n}\mathbb E\hat\epsilon_i^2$$

  • (1). $\text{Var}(Y_i)=\text{Var}(\epsilon_i)=\sigma^2$
  • (2). $$\hat\beta_1=\sum_{i=1}^{n}\frac{(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}y_i$$ then $$\begin{align} \hat\beta_0&=\bar{y}-\hat\beta_1\bar{x}\\ &=\sum\frac{1}{n}y_i-\frac{\bar{x}\sum(x_i-\bar{x})y_i}{\sum(x_i-\bar{x})^2}\\ &=\sum\left[\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}\right]y_i \end{align}$$ For $$c_i=\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{\sum(x_i-\bar{x})^2},$$ $\sum c_i=1,\sum c_ix_i=0$. Then $$\mathbb E(\hat\beta_0)=\mathbb E(\sum c_iY_i)=\sum c_i(\beta_0+\beta_1x_i)=\beta_0$$

$$\begin{align} \text{Var}(\hat\beta_0)&=\text{Var}(\sum c_iY_i)\\ &=\sum c_i^2\text{Var}(Y_i)\\ &=\sigma^2\sum c_i^2\\ &=\sigma^2\sum\left[\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}\right]^2\\ &=\sigma^2\left[\sum\frac{1}{n^2}+\frac{\sum\bar{x}^2(x_i-\bar{x})^2}{\left(\sum(x_i-\bar{x})^2\right)^2}\right]\\ &=\sigma^2\left[\frac{1}{n}+\frac{\bar{x}^2}{\sum(x_i-\bar{x})^2}\right]\\ &=\sigma^2\left[\frac{\sum(x_i-\bar{x})^2+n\bar{x}^2}{n\sum(x_i-\bar{x})^2}\right]\\ &=\sigma^2\left[\frac{\sum x_i^2}{n\sum(x_i-\bar{x})^2}\right]\\ &=\sigma^2\left[\frac{\sum x_i^2}{nS_{xx}}\right]\\ \end{align}$$

  • (3). $$\text{Var}\hat\beta_1=\sigma^2\sum_{i=1}^{n}(x_i-\bar x)^2/S_{xx}^2=\frac{\sigma^2}{\sum_{i=1}^{n}(x_i-\bar x)^2}$$

  • (4). Write $\hat\beta_1=\sum d_iy_i$ where $d_i=\frac{x_i-\bar{x}}{\sum(x_i-\bar{x})^2}=\frac{x_i-\bar{x}}{S_{xx}}$, $$\begin{align} \text{Cov}(\hat\beta_0,\hat\beta_1)&=\text{Cov}\left(\sum c_iY_i,\sum d_iY_i\right)\\ &=\left(\sum_{i=1}^{n}c_id_i\right)\sigma^2\\ &=\sigma^2\sum_{i=1}^{n}\left(\left[\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}\right]\frac{x_i-\bar{x}}{\sum(x_i-\bar{x})^2}\right)\\ &=\sigma^2\sum_{i=1}^{n}\left(\left[\frac{\sum(x_i-\bar{x})^2-n\bar{x}(x_i-\bar{x})}{n\sum(x_i-\bar{x})^2}\right]\frac{x_i-\bar{x}}{\sum(x_i-\bar{x})^2}\right)\\ &=\sigma^2\left(\left[\frac{\sum(x_i-\bar{x})\sum(x_i-\bar{x})^2-n\bar{x}\sum(x_i-\bar{x})^2}{n\sum(x_i-\bar{x})^2}\right]\frac{1}{\sum(x_i-\bar{x})^2}\right)\\ &=\sigma^2\left(\frac{\sum(x_i-\bar{x})-n\bar{x}}{n\sum(x_i-\bar{x})^2}\right)\\ &=\sigma^2\left(\frac{-\bar{x}}{\sum(x_i-\bar{x})^2}\right)\\ \end{align}$$

  • (5). $$\begin{align} \text{Cov}(Y_i, \hat\beta_0)&=\text{Cov}\left(Y_i, \sum c_iY_i\right)\\ &=\left[\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{\sum(x_i-\bar{x})^2}\right]\sigma^2\\ \end{align}$$

  • (6). $$\begin{align} \text{Cov}(Y_i, \hat\beta_1)&=\text{Cov}\left(Y_i, \sum d_iY_i\right)\\ &=\frac{x_i-\bar{x}}{\sum(x_i-\bar{x})^2}\sigma^2\\ \end{align}$$

Thus $$\begin{align} \mathbb E\hat\sigma^2&=\frac{1}{n}\sum_{i=1}^{n}\mathbb E \hat\epsilon_i^2\\ &=\frac{1}{n}\sum_{i=1}^{n}\left[\left(1+\frac{\sum x_i^2}{n\sum(x_i-\bar{x})^2}+\frac{x_i^2}{\sum_{i=1}^{n}(x_i-\bar x)^2}-\frac{2}{n}-\frac{2(x_i-\bar{x})^2}{\sum(x_i-\bar{x})^2}-\frac{2x_i\bar{x}}{\sum(x_i-\bar{x})^2}\right)\sigma^2\right]\\ &=\left(\frac{n-2}{n}+\frac{\sum x_i^2}{n\sum(x_i-\bar{x})^2}+\frac{\sum x_i^2}{n\sum_{i=1}^{n}(x_i-\bar x)^2}-\frac{2\sum(x_i-\bar{x})^2}{n\sum(x_i-\bar{x})^2}-\frac{2\sum x_i\bar{x}}{n\sum(x_i-\bar{x})^2}\right)\sigma^2\\ &=\left(\frac{n-2}{n}+0\right)\sigma^2\\ &=\frac{n-2}{n}\sigma^2\\ \end{align}$$

I solved this problem at last, but I would like to leave this question here.

$\endgroup$
1