1
$\begingroup$

I am trying to understand this answer by @whuber to this question of how to get the standard errors of the regression estimators. @whuber says the following:

Variance of the slope estimate

The Ordinary Least Squares estimate of the slope in these new units is simply the average product of the $\xi_i$ and $y_i,$

$$\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i.\tag{2}$$

I can see that we can write the standardised value as

$$\dfrac{x_i - \overline{x}}{\sigma_x} = \zeta_i.$$

So the slope in the new units, $\beta_1 \zeta_i$, is the old slope, $\beta_1$, scaled by the standard deviation of all the $x$, $\sigma_x$. With that said, I still don't see where this idea that $\hat{\beta_1 \sigma_x} = \dfrac{1}{n} \sum\limits_{i = 1}^n \zeta_i y_i$ is coming from.

@whuber then says that $\hat{\beta_1 \sigma_x} = \dfrac{1}{n} \sum\limits_{i = 1}^n \zeta_i y_i$ comes from the normal equation of least squares. Following the link in @whuber's answer here to this question, I fully write out a small example of $(X^T X)\beta = X^T Y$, where $n = 2$ and $p = 2$ (see here), to get

$$\begin{bmatrix} 2 \beta_0 + (x_{11} + x_{21})\beta_1 + (x_{12} + x_{22}) \beta_2 \\ (x_{11} + x_{21})\beta_0 + (x_{11}^2 + x_{21}^2)\beta_1 + (x_{11}x_{12} + x_{21}x_{22})\beta_2 \\ (x_{12} + x_{22})\beta_0 + (x_{11}x_{12} + x_{21}x_{22})\beta_1 + (x_{12}^2 + x_{22}^2)\beta_2 \end{bmatrix} = \begin{bmatrix} (\beta_0 + \beta_1 x_{11} + \beta_2 x_{12}) + (\beta_0 + \beta_1 x_{21} + \beta_2 x_{22}) \\ x_{11} (\beta_0 + \beta_1 x_{11} + \beta_2 x_{12}) + x_{21}(\beta_0 + \beta_1 x_{21} + \beta_2 x_{22}) \\ x_{12}(\beta_0 + \beta_1 x_{11} + \beta_2 x_{12}) + x_{22}(\beta_0 + \beta_1 x_{21} + \beta_2 x_{22}) \end{bmatrix}$$

This seems to be correct. But, even so, I don't see how this clarifies $\hat{\beta_1 \sigma_x} = \dfrac{1}{n} \sum\limits_{i = 1}^n \zeta_i y_i$.

So how do we get that $\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i$?

$\endgroup$
2
+50
$\begingroup$

It stems from the expression for the estimate of the slope-coefficient of the simple linear regression

$$\hat{\beta}_1 = \dfrac{\sum\limits_{i = 1}^{n}(x_i - \overline{x})(y_i - \overline{y})}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}$$

Or in the new coordinates using $\xi_i$ instead of $x_i$

$$\widehat{\beta_1 \sigma_x} = \dfrac{\sum\limits_{i = 1}^{n}(\xi_i - \overline{\xi})(y_i - \overline{y})}{\sum\limits_{i = 1}^{n}(\xi_i - \overline{\xi})^2}$$

And these $\xi_i$ where cleverly defined such that

$$\begin{array}{rcl} \sum\limits_{i = 1}^{n}(\xi_i - \overline{\xi})^2 &=& n \\ \bar \xi &=& 0 \end{array}$$

If you fill those into the expression for $ \widehat{\beta_1 \sigma_x}$ then you get to the simpler expression for the estimate

$$\begin{array}{} \widehat{\beta_1 \sigma_x} &=& \dfrac{\sum\limits_{i = 1}^{n}(\xi_i - \overline{\xi})(y_i - \overline{y})}{\sum\limits_{i = 1}^{n}(\xi_i - \overline{\xi})^2} \\ && \text{filling in / substituting} \\ &=& \frac{\sum\limits_{i = 1}^{n}\xi_i(y_i - \overline{y})}{n} \\ && \text{splitting the product with a sum into a sum of two products} \\ &=& \frac{{\sum\limits_{i = 1}^{n}\xi_i y_i } + \sum\limits_{i = 1}^{n}- \xi_i \overline{y}}{n} \\ && \text{move term $\bar{y}$ outside the sum (it's not dependent on $i$)} \\ &=& \frac{{\sum\limits_{i = 1}^{n}\xi_i y_i } - \overline{y} \sum\limits_{i = 1}^{n} \xi_i}{n} \\&& \text{the sum of $\xi_i$ is zero} \\ &=& \frac{\sum\limits_{i = 1}^{n}\xi_i y_i }{n} \\&& \text{move the division by $n$} \\ &=& \sum\limits_{i = 1}^{n}\frac{\xi_i}{n} y_i \end{array}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.