8
$\begingroup$

Title. I'd like to understand the intuition behind how regression coefficients are calculated, and why $$\frac{cov(x,y)}{var(x)}$$ gives a regression coefficient for dependent variable y and independent variable x? This feels like an elementary question but I have looked around the internet and can't anything answering this question specifically.

$\endgroup$
5
  • 1
    $\begingroup$ It's important to note that $cov$ and $var$ in your posted formula are their sample counterparts: $var(x) = \frac{1}{n-1}\sum_{i=1}^n \left(x_i - \bar{x}\right)^2$ and $cov(x,y) = \frac{1}{n-1}\sum_{i=1}^n \left(x_i - \bar{x}\right)\left(y_i - \bar{y}\right)$ $\endgroup$
    – user277126
    Dec 12, 2021 at 22:07
  • $\begingroup$ @user277126 Is that important? For the population, we would compute the regression coefficient in the same way. $\frac{1}{n-1}$ occurs in both numerator and denominator, we could replace it by any constant, e.g., $\frac{1}{n}$. $\endgroup$ Dec 12, 2021 at 23:18
  • $\begingroup$ I only meant to differentiate them between the meaning of these operators in statistics $\endgroup$
    – user277126
    Dec 12, 2021 at 23:29
  • $\begingroup$ @user277126 Thanks! But if we take expectations, then the fraction still gives the regression parameter? $\endgroup$ Dec 13, 2021 at 0:38
  • $\begingroup$ stats.stackexchange.com/… discloses many posted explanations. Perhaps the most straightforward ones rely on the most basic characterizations of the situations: namely, the correlation coefficient is defined as the mean product and the regression slope, as it turns out, also is the mean product when the two variables have first been separately standardized. $\endgroup$
    – whuber
    Dec 13, 2021 at 18:48

3 Answers 3

13
$\begingroup$

Why $$\frac{cov(x,y)}{var(x)}$$ gives a regression coefficient for dependent variable y and independent variable x?

A linear regression coefficient tells us: If predictor variable $x$ increases by 1, what is the expected increase in outcome variable $y$?

The answer to this question depends in large part on the scales on which $x$ and $y$ are measured. E.g., if $x$ is a measure of length, imagine measuring in millimeters or centimeters; the variance of measurements in millimeters will be $10^2$ times the variance of the same measurements in centimeters; the covariance will be multiplied by 10. Note, $cov(x,y)$ is determined by three things:

  1. the linear association between $x$ and $y$;
  2. the scale of $x$;
  3. the scale of $y$.

Because of 2) and 3), I would call the covariance an unstandardized measure of association. Its value is difficult to interpret, because what would be a large and what would be a small value depends on the scales of $x$ and $y$. The correlation coefficient, however, gives us a standardized measure of association: It is 'corrected' for the scales on which $x$ and $y$ are measured:

$$cor(x,y)=\frac{cov(x,y)}{\sqrt{var(x) \cdot var(y)}}$$

The correlation coefficient tells us: If $x$ increases by $\sqrt{var(x)}$, how many $\sqrt{var(y)}$s will outcome $y$ increase? Thus, with a correlation coefficient of 1, an increase of 1 SD in $x$ is associated with an increase of 1 SD in $y$.

Now, the regression coefficient quantifies the expected increase in $y$, when $x$ increases by 1. We thus need to 'correct' the covariance between $x$ and $y$ for the scale of $x$. We can do that by simply dividing:

$$\frac{cov(x,y)}{var(x)}$$

Note that if we would 'reverse' the problem, and ask the question: If $y$ increases by 1, what is the expected increase in $x$? We can compute the answer as follows:

$$\frac{cov(x,y)}{var(y)}$$

$\endgroup$
6
$\begingroup$

In simple linear regression we are dealing with the model \begin{eqnarray*} y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \mbox{for} \quad i=1,\cdots,n. \end{eqnarray*}

Let $\boldsymbol{y} = \left(y_1, \cdots, y_n\right)^{\prime}$, $\boldsymbol{x} = \left(x_1, \cdots, x_n\right)^{\prime}$, and $\boldsymbol{\beta} = \left(\beta_0, \beta_1\right)$. Moreover, the design matrix is $\boldsymbol{X} = \left[\boldsymbol{1}_n, \boldsymbol{x} \right]$. It may be shown that the MLE of $\boldsymbol{\beta}$ is \begin{eqnarray*} \widehat{\boldsymbol{\beta}} &=& \left(\boldsymbol{X}^{\prime}\boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \\ &=& \frac{1}{n\sum_{i=1}^n x_i^2 - n^2 \bar{x}^2} \begin{pmatrix} \sum_{i=1}^n x_i^2 & -n\bar{x} \\ -n\bar{x} & n \end{pmatrix} \begin{pmatrix} n \bar{y} \\ \sum_{i=1}^n x_iy_i \end{pmatrix}. \end{eqnarray*} Clearly, the second term, $\widehat{\beta}_1$, is given by \begin{eqnarray*} \widehat{\beta}_1 &=& \frac{n\sum_{i=1}^n x_iy_i - n^2 \bar{x}\bar{y}}{n\sum_{i=1}^n x_i^2 - n^2 \bar{x}^2} \\ &=& \frac{\frac{\sum_{i=1}^n x_iy_i - n \bar{x}\bar{y}}{n-1}}{\frac{\sum_{i=1}^n x_i^2 - n \bar{x}^2}{n-1}} \\ &=& \frac{cov(x,y)}{var(x)}, \end{eqnarray*} where $var$ and $cov$ represent their sample counterparts.

$\endgroup$
2
$\begingroup$

The first thing to mention is that the $\frac{cov(x,y)}{var(x)}$ will only be the $b_1$ in the case of this regression: $y = b_0 + b_1x$, that is, when we only have two variables. If we have a third, for example, this breaks down as you can test with the R code below:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)
cov(X,Y)/var(X)

Running the code above you will get the same value from the last two lines, which is $0.9882329$. But if you run the code below:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
Z <- X + rnorm(N)
lm(Y~X + Z)
cov(X,Y)/var(X)

You will get $0.9887337$ for $b_1$ and $0.9882329$ for $\frac{cov(X,Y)}{var(X)}$. And even though these values are close here, this is not always the case. If I had Z <- Y + rnorm(N), you would see $0.47370$ for $b_1$.

Now for your question. The variance measures how spread are the data points of a variable when compared to its mean. The covariance, in a way, measures if the spread in variable X follows the spread in variable Y. See the example below.

$cov(X,Y) = E[(X-E[X])(Y-E[Y])]$

Let's look at a data point at a time. We estimated that the mean for $X$ is $5$ and for $Y$ it is $10$. The first value we have for $X$ in the dataset is $20$, which is above its mean ($20\gt 5$). The first value we have for $Y$ is $0$, which is below its mean ($0 \lt 10$). Let's plug these values in the equation for covariance.

$(X-E[X])(Y-E[Y]) = (20-5)(0-10) = (15)(-10) = -150$.

Another data point now. $6$ for $X$ and $5$ for $Y$. Again, we will get a negative value.

$(X-E[X])(Y-E[Y]) = (6-5)(5-10) = (1)(-5) = -5$

If we keep doing this, and keep seeing this behaviour, it's easy to see that at the end we will have a negative covariance. $X$ values will to some extent be frequently above its mean, while $Y$ values will to some extent be frequently below its mean. Knowing that the Pearson correlation coefficient (PCC) is the standardised covariance ($ r = \frac{cov(X,Y)}{\sigma X \sigma Y}$), we would then equally get a negative value, that is, a negative correlation.

What does a negative correlation tell you right away? If I don't tell you the value of the PCC, the only thing the sign gives you is the "orientation" of the slope (increasing or decreasing). What's the $b_1$ coefficient in $y = b_0 + b_1x$? The slope! :-)

Based on that, I assume it's clear that the covariance of $X$ and $Y$ is proportional to the slope ($b_1$) of the regression line of $Y$, as dependent variable, and $X$ as independent variable.

However, there is a catch here. If you try to do a regression swapping who is the independent variable and who is the dependent one, you will realize the coefficients change! See the code below.

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)$coefficients
lm(X~Y)$coefficients

$b_1$ for the first regression will be $0.98823293$, as we already saw, but for the second regression we will get $0.50075232$. Based on what I said about $cov(X,Y)$, someone may think of trying $cov(Y,X)$ but covariance is symmetric, that is, $cov(X,Y) = cov(Y,X)$. If you divide this by the variance of the independent variable, you will get the right value. The code below shows this:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)$coefficients
lm(X~Y)$coefficients
cov(Y,X)
cov(X,Y)

cov(X,Y)/var(X)
cov(X,Y)/var(Y)

For the derivation of the least squares estimators of the slope and intercept in linear regression, this is a complete and slow-paced video.

$\endgroup$
2
  • $\begingroup$ You mentioned covariance gives the orientation of the slope. Which means we find covariance in calculating slope only to find whether the slope is negative or positive, right? But, why do we divide covariance by variance of the independent variable? I can't relate how slope is related to how much x values are spread out? What does dividing with variance gives to slope? $\endgroup$
    – F.C. Akhi
    Dec 18, 2021 at 21:24
  • $\begingroup$ Yes. The part abou dividing by the variance is answered by Marjolein Fokkema: "Now, the regression coefficient quantifies the expected increase in 𝑦, when 𝑥 increases by 1. We thus need to 'correct' the covariance between 𝑥 and 𝑦 for the scale of 𝑥. We can do that by simply dividing:" [the covariance of X and Y by the variance of X, the independent variable]. $\endgroup$ Dec 19, 2021 at 0:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.