3
$\begingroup$

I'm reading a pdf document about simple linear regression, and it gave an expression of the least square estimator $\hat{\beta}_1 = \beta_1 + \frac{\sum^n_{i=0} \epsilon_i (x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$, but I'm confused on how they get to this answer.

I only know that $\hat{\beta}_1 = \frac{\sum^n_{i=0} y_i(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2} = \frac{\sum^n_{i=0} (\hat{y_i} + \epsilon_i)(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$. Does it mean $\beta_1 = \frac{\sum^n_{i=0} \hat{y_i}(x_i - \bar{x})}{\sum^n_{i=0} (x_i - \bar{x})^2}$? Why is that?

$\endgroup$

2 Answers 2

4
$\begingroup$

This is essentially an algebraic exercise. Noting the $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ you then have:

$$\begin{align} \sum_i y_i(x_i - \bar{x}) &= \sum_i (\beta_0 + \beta_1 x_i + \epsilon_i)(x_i - \bar{x}) \\[6pt] &= \beta_0 \sum_i (x_i - \bar{x}) + \beta_1 \sum_i x_i (x_i - \bar{x}) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_0 (n\bar{x} - n\bar{x}) + \beta_1 \Big( \sum_i x_i^2 - n \bar{x}^2 \Big) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_1 \Big( \sum_i x_i^2 - 2 n \bar{x}^2 + n \bar{x}^2 \Big) + \sum_i \epsilon_i (x_i - \bar{x}) \\[6pt] &= \beta_1 \sum_i ( x_i - \bar{x})^2 + \sum_i \epsilon_i (x_i - \bar{x}). \\[6pt] \end{align}$$

Substituting into your initial expression for the estimated coefficient, you get:

$$\hat{\beta}_1 = \frac{\sum_i y_i(x_i - \bar{x})}{\sum_i ( x_i - \bar{x})^2} = \beta_1 + \frac{\sum_i \epsilon_i (x_i - \bar{x})}{\sum_i ( x_i - \bar{x})^2}.$$

(Also, usually the convention is that you would have observations $i=1,...,n$, so I'm not sure why you're summing over $i=0,...,n$. In my working I sum over the former range.)

$\endgroup$
5
$\begingroup$

The least squares normal equations are

$$\mathbf X^\mathsf T\mathbf X\mathbf b = \mathbf X^\mathsf T\mathbf y \tag 1$$ and thus $$\text{(OLS)}: ~~\mathbf b =\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\mathbf y. \tag 2\label 2$$

Putting $\mathbf y =\mathbf X \boldsymbol\beta+\boldsymbol\varepsilon $ in \eqref{2} yields

\begin{align}\mathbf b &=\left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T(\mathbf X \boldsymbol\beta+\boldsymbol\varepsilon)\\ &= \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\mathbf X \boldsymbol\beta+ \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon\\&= \boldsymbol\beta + \left(\mathbf X^\mathsf T\mathbf X\right)^{-1}\mathbf X^\mathsf T\boldsymbol\varepsilon.\tag 3\label 3\end{align}

For the two-variable case, $\mathbf X = \begin{bmatrix}\mathbf 1 & \mathbf x\end{bmatrix}; $ so \begin{align}\mathbf X^\mathsf T\mathbf X &= \begin{bmatrix}\mathbf 1 ^\mathsf T\\\mathbf x^\mathsf T\end{bmatrix} \begin{bmatrix}\mathbf 1 & \mathbf x\end{bmatrix} \\ &=\begin{bmatrix}n &\sum x_i\\\sum x_i& \sum x_i^2\end{bmatrix}.\tag 4\end{align}

And $$ \mathbf X^\mathsf T\boldsymbol\varepsilon =\begin{bmatrix}\sum \varepsilon_i\\ \sum x_i\varepsilon_i\end{bmatrix}.\tag 5$$

OP can complete the rest of the bookkeeping calculation to reach to the desired expression in league with \eqref{3}.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.