3
$\begingroup$

Can someone provide an intuitive explanation of why the OLS regression estimate, of y=a+bx, b have the form b=cov(x,y)/V(x).

How intuitively are the covariance and variance related in this?

$\endgroup$
1
  • 1
    $\begingroup$ "Intuitive" is in the eye of the beholder. This result is so nonintuitive that it wasn't discovered until a few hundred years ago and its importance was established only 140 years ago. If you view geometric arguments as intuitive, then take a look at stats.stackexchange.com/a/71303/919. $\endgroup$
    – whuber
    Oct 6, 2018 at 15:31

2 Answers 2

1
$\begingroup$

First, think about normalized version, i.e. $X$ and $Y$ have unit variances. Then, the term you describe becomes $\rho_{XY}$, which is the Pearson correlation coefficient. And, we have $y=\rho x+b$. This is totally intuitive since $y$ and $x$ are directly bounded with their correlation coefficient.

Now, think about the term you write: $$\frac{cov(X,Y)}{\sigma_X^2}=\rho\frac{\sigma_y}{\sigma_x} \rightarrow y=\rho\frac{\sigma_y}{\sigma_x}x+a$$

$y$ and $x$ are still bounded with their correlation coefficients. But, we also calibrate the variances. $x$ is first made unit variance by dividing it with $\sigma_x$, then calibrated upto the variance of $y$ with multiplying with $\sigma_y$. So, it is like (not the same) dividing by the range of $x$ and multiplying with the range of $y$.

$\endgroup$
0
$\begingroup$

I think a couple of plots will easily persuade you intuitively.

As gunes said, the regression line can be expressed as:

$$y= r \frac{\sigma_y}{\sigma_x}+a$$

The first thing which should be quite plausible is that (for positive correlation) the larger the correlation the greater the slope of the regression line

In order to see why the SD must also play a role for the slope of the regression line consider the following example:

Imagine that $Y$ contains values of son`s height which we want to predict by using the father's weight. We have multiple ways to measure height. We could e.g. measure father's and son's height in cm, and imagine that we obtain the equation:

$\sigma_x=1, \sigma_y=1, r=0.9, \mu_x=\mu_y=1.80 $ we would therefore obtain the regression line:

$$y \text{ (in cm)}= 0.9x\text{ (in cm)}+0.18$$ and we interprete this as "1 cm more in father's height will yield 0.9 cm more in son's height"

however we could also measure the height of the fathers in cm and and the sons in mm: the equation is then:

$$y \text{ (in mm)}= 90x\text{ (in cm)}+0.18$$

and we interprete this: "one cm more in father's height yill yield 9 mm more in son's height"

or the other way around:

$$y \text{ (in cm)}= 0.09x\text{ (in mm)}+0.18$$

We see (at least for the example), that the scale of the data (whether we measure in cm or mm) has to influence the slope of the regression line

Comparing the upper left with the upper right plot illustrates the idea that higher correlation yields a larger slope in the regression line.

The data from the other three (all but the upper left) plots is from 2d-Normals with the same correlation, but with different SDs

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.