3
$\begingroup$

I am looking at my statistics book and an article online for linear regression and was wondering if anyone can verify that these two equations are entirely different. Consider the equation $\hat{y} = ax + b$

In my book, a and b are :

$a = \frac{r \cdot S_{y}}{S_{x}}$

$b = \bar{y} - a\bar{x}$

$r = \sum \frac{(x_{i} - \bar{x})(y_{i} -\bar{y})}{S_{x}S_{y}(n-1)}$

$\displaystyle S_{y} = \sqrt{ \frac{\sum (y_i - \bar{y})^{2}}{(n-1)} }$

From one online article, a and b are:

$\displaystyle a = \frac{n \sum x_{i}y_{i} - \sum x_{i} \sum y_{i}}{n \sum x^2_{i} - (\sum x_{i})^2}$

$b = \bar{y} - a\bar{x}$.

The a from the online article vaguely looks like covariance in the numerator and the denominator looks like variance but for only one random variable, not two. Can someone explain the discrepancy (if there are any) and construct an argument for my book's choice? I can understand the second formulation mainly because it comes from setting partial derivatives to zero to minimize an objective function and then finding the coefficients a and b.

$\endgroup$
1
  • $\begingroup$ please define $r$, $S_x$ and $S_y$. $\endgroup$
    – Memming
    Oct 20, 2013 at 22:37

2 Answers 2

2
$\begingroup$

They are mainly equivalent. First rewrite $r$ as $r=\dfrac{S_{xy}}{(n-1)S_xS_y}$, where $S_{xy}=\sum(x_i-\bar{x}).(y_i-\bar{y})$. Now plug in $r$ as above in $a$ from your book, we have: $a=\dfrac{r.S_y}{S_x}=\dfrac{S_{xy}}{(n-1)S_xS_y}.\dfrac{S_y}{S_x}=\dfrac{S_{xy}}{S_x^2(n-1)}$. But from definition of $S_y$ from your book, we can have $S_x=\sqrt{\dfrac{\sum (x_i-\bar{x})^2}{n-1}}=\sqrt{\dfrac{S_{xx}}{n-1}}$. Now square both sides and use cross multiplication to have: $S_{xx}=S_x^2.(n-1)$. Now replace this last equality in the denominator of $a$ we found before to get: $a=\dfrac{S_{xy}}{S_{xx}}$. Now work on $a$ from online. Note that $S_{xy}=\sum x_iy_i-\bar{y}\sum x_i-\bar{x}\sum y_i+n\bar{y}\bar{x}=\sum x_iy_i-n\bar{y}\bar{x}$. Hence we have $nS_{xy}=n\sum x_iy_i-n^2\bar{y}\bar{x}=n\sum x_iy_i-\sum x_i \sum y_i.$ So what you have in the numerator of $a$ from online is actually $nS_{xy}$. Now work on the denominator of $a$ and factor out $n$ to get $n\sum x_i^2-(\sum x_i)^2=n\Big(\sum x_i^2-\dfrac{(\sum x_i)^2}{n}\Big )=n\Big(\sum x_i^2-\dfrac{n^2\bar{x}^2}{n}\Big )=n\Big(\sum x_i^2-n\bar{x}^2\Big)=nS_{xx}.$ So what you have in the denominator of $a$ is $nS_{xx}$. Therefore, $a$ from online is actually $a=\dfrac{nS_{xy}}{nSxx}=\dfrac{S_{xy}}{Sxx}$ that is equal to the $a$ from your book. Cheers :)

$\endgroup$
0
2
$\begingroup$

I don't know what online article you looked it (maybe you could link it?), but as far as I can tell, the author there assumes that $x$ and $y$ are centered about their means. Under this assumption, the two formulae for $a$ that you posted do agree.

To see this, first note that your first $a$ formula can written as $$ a = \frac{r \cdot S_y}{S_x} = \frac{{\rm cov}(y,x) \cdot S_y}{(S_y \cdot S_x) \cdot S_x} = \frac{{\rm cov}(y,x)}{S_x^2}, $$ with ${\rm cov}$ referring to the covariance.

Now if you take the second $a$ formula and assume the variables are centered, then the simple sum terms drop out (because then both variables separately sum to 0). So it just reduces to $$ a = \frac{\sum x_{i}y_{i}}{\sum x^2_{i}} = \frac{\sum x_{i}y_{i} / (n-1)}{\sum x^2_{i} / (n-1)} = \frac{{\rm cov}(y,x)}{S_x^2}, $$ which matches the first formula. So they coincide if the variables are centered, but not in general. Maybe they mention this assumption in the online article you referred to.

Edit: I suggested that they were not equivalent in general, only in the special case where variables are centered. But @Stat has apparently shown that they are equivalent in general. So I will just leave this answer up in case you find considering this special case to be illuminating :)

$\endgroup$
5
  • $\begingroup$ If you think @stat's answer is good, you might upvote it... ;-) $\endgroup$ Oct 21, 2013 at 2:14
  • $\begingroup$ @gung Haha, yes of course, thanks for the reminder $\endgroup$ Oct 21, 2013 at 2:18
  • $\begingroup$ Could you explain how do they just drop out and give you 0? Here's the article you asked: ensign.editme.com/linearregression $\endgroup$
    – Person
    Oct 21, 2013 at 2:38
  • $\begingroup$ I'm guessing you're saying that $x$ and $y$ have the same mean? $\endgroup$
    – Person
    Oct 21, 2013 at 2:56
  • $\begingroup$ @Guest If a variable has a mean of 0 (this is what it means for a variable to be "centered" about its mean), then its sum must also be 0. After all, the mean of a variable is just computed as its sum divided by something. So if the mean is 0, the sum must also be 0. Make sense? $\endgroup$ Oct 21, 2013 at 2:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.