Expression of $\hat{\beta}_1$ in term s of $\beta_1$ in simple linear regression

Question

I'm reading a pdf document about simple linear regression, and it gave an expression of the least square estimator $\hat{\beta}_1 = \beta_1 + \frac{\sum^n_{i=0} \epsilon_i (x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$, but I'm confused on how they get to this answer.

I only know that $\hat{\beta}_1 = \frac{\sum^n_{i=0} y_i(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2} = \frac{\sum^n_{i=0} (\hat{y_i} + \epsilon_i)(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$. Does it mean $\beta_1 = \frac{\sum^n_{i=0} \hat{y_i}(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$? Why is that?

Answer 1

This is essentially an algebraic exercise. Noting the $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ you then have:

$$\begin{align} \sum_i y_i(x_i - \bar{x}) &= \sum_i (\beta_0 + \beta_1 x_i + \epsilon_i)(x_i - \bar{x}) \\[6pt] &= \beta_0 \sum_i (x_i - \bar{x}) + \beta_1 \sum_i x_i (x_i - \bar{x}) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_0 (n\bar{x} - n\bar{x}) + \beta_1 \Big( \sum_i x_i^2 - n \bar{x}^2 \Big) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_1 \Big( \sum_i x_i^2 - 2 n \bar{x}^2 + n \bar{x}^2 \Big) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_1 \sum_i ( x_i - \bar{x})^2 + \sum_i \epsilon_i (x_i - \bar{x}). \\[6pt] \end{align}$$

Substituting into your initial expression for the estimated coefficient, you get:

$$\hat{\beta}_1 = \frac{\sum_i y_i(x_i - \bar{x})}{\sum_i ( x_i - \bar{x})^2} = \beta_1 + \frac{\sum_i \epsilon_i (x_i - \bar{x})}{\sum_i ( x_i - \bar{x})^2}.$$

(Also, usually the convention is that you would have observations $i=1,...,n$, so I'm not sure why you're summing over $i=0,...,n$. In my working I sum over the former range.)

Answer 2

The least squares normal equations are

$$\mathbf X^\mathsf T\mathbf X\mathbf b = \mathbf X^\mathsf T\mathbf y \tag 1$$ and thus $$\text{(OLS)}: ~~\mathbf b =\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\mathbf y. \tag 2\label 2$$

Putting $\mathbf y =\mathbf X \boldsymbol\beta+\boldsymbol\varepsilon $ in \eqref{2} yields

\begin{align}\mathbf b &=\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T(\mathbf X \boldsymbol\beta+\boldsymbol\varepsilon)\\ &= \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\mathbf X \boldsymbol\beta+ \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon\\&= \boldsymbol\beta + \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon.\tag 3\label 3\end{align}

For the two-variable case, $\mathbf X = \begin{bmatrix}\mathbf 1 & \mathbf x\end{bmatrix}; $ so \begin{align}\mathbf X^\mathsf T\mathbf X &= \begin{bmatrix}\mathbf 1 ^\mathsf T\\\mathbf x^\mathsf T\end{bmatrix} \begin{bmatrix}\mathbf 1 & \mathbf x\end{bmatrix} \\ &=\begin{bmatrix}n &\sum x_i\\\sum x_i& \sum x_i^2\end{bmatrix}.\tag 4\end{align}

And $$ \mathbf X^\mathsf T\boldsymbol\varepsilon =\begin{bmatrix}\sum \varepsilon_i\\ \sum x_i\varepsilon_i\end{bmatrix}.\tag 5$$

OP can complete the rest of the bookkeeping calculation to reach to the desired expression in league with \eqref{3}.