In An Introduction to Statistical Learning (James et al.), in section 3.7 exercise 5, it states that the formula for $\hat{\beta}_1$ assuming linear regression without an intercept is $$\hat{\beta}_1 = \dfrac{\displaystyle\sum\limits_{i=1}^{n}x_iy_i}{\displaystyle \sum\limits_{i=1}^{n}x_i^2}\text{,}$$ where $\hat{\beta}_0 = \bar{y}-\hat{\beta}_1\bar{x}$ and $\hat{\beta}_1 = \dfrac{\displaystyle S_{xy}}{S_{xx}}$ are the usual estimates under OLS for simple linear regression ($S_{xy} = \displaystyle\sum\limits_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})$).

This is not the actual exercise; I am merely wondering how to derive the equation. Without using matrix algebra, how do I derive it?

My attempt: with $\hat{\beta}_0 = 0$, we have $\hat{\beta}_1 = \dfrac{\bar{y}}{\bar{x}} = \dfrac{S_{xy}}{S_{xx}}$.

After some algebra, it can be shown that $\displaystyle S_{xy} = \sum\limits_{i=1}^{n}x_i^2 - n\bar{x}\bar{y}$ and $\displaystyle S_{xx} = \sum\limits_{i=1}^{n}x_i^2 - n\bar{x}^2$. From here, I'm stuck.

  • 1
    $\begingroup$ The formula is immediate from the geometric interpretation of least squares, using $x/||x||$ as a "matcher" for $y$ and recognizing the formula for $(\hat\beta_1)x$ as being $(y\cdot (x/||x||))x/||x||$. $\endgroup$ – whuber Jul 2 '15 at 21:02
  • $\begingroup$ @whuber : Rather than writing $x/||x||,$ I would write $x/\|x\|.$ If that's not conspicuous enough for you, consider the typographical difference between $||x|| ||y||,$ coded as ||x|| ||y||, and $\|x\|\|y\|,$ coded as \|x\|\|y\|. $\qquad$ $\endgroup$ – Michael Hardy Feb 3 at 18:31

This is straightforward from the Ordinary Least Squares definition. If there is no intercept, one is minimizing $R(\beta) = \sum_{i=1}^{i=n} (y_i- \beta x_i)^2$. This is smooth as a function of $\beta$, so all minima (or maxima) occur when the derivative is zero. Differentiating with respect to $\beta$ we get $-\sum_{i=1}^{i=n} 2(y_i- \beta x_i)x_i$. Solving for $\beta$ gives the formula.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.