Why is a regression coefficient covariance/variance

Question

Title. I'd like to understand the intuition behind how regression coefficients are calculated, and why $$\frac{cov(x,y)}{var(x)}$$ gives a regression coefficient for dependent variable y and independent variable x? This feels like an elementary question but I have looked around the internet and can't anything answering this question specifically.

Answer 1

Why $$\frac{cov(x,y)}{var(x)}$$ gives a regression coefficient for dependent variable y and independent variable x?

A linear regression coefficient tells us: If predictor variable $x$ increases by 1, what is the expected increase in outcome variable $y$?

The answer to this question depends in large part on the scales on which $x$ and $y$ are measured. E.g., if $x$ is a measure of length, imagine measuring in millimeters or centimeters; the variance of measurements in millimeters will be $10^2$ times the variance of the same measurements in centimeters; the covariance will be multiplied by 10. Note, $cov(x,y)$ is determined by three things:

1. the linear association between $x$ and $y$;
2. the scale of $x$;
3. the scale of $y$.

Because of 2) and 3), I would call the covariance an unstandardized measure of association. Its value is difficult to interpret, because what would be a large and what would be a small value depends on the scales of $x$ and $y$. The correlation coefficient, however, gives us a standardized measure of association: It is 'corrected' for the scales on which $x$ and $y$ are measured:

$$cor(x,y)=\frac{cov(x,y)}{\sqrt{var(x) \cdot var(y)}}$$

The correlation coefficient tells us: If $x$ increases by $\sqrt{var(x)}$, how many $\sqrt{var(y)}$s will outcome $y$ increase? Thus, with a correlation coefficient of 1, an increase of 1 SD in $x$ is associated with an increase of 1 SD in $y$.

Now, the regression coefficient quantifies the expected increase in $y$, when $x$ increases by 1. We thus need to 'correct' the covariance between $x$ and $y$ for the scale of $x$. We can do that by simply dividing:

$$\frac{cov(x,y)}{var(x)}$$

Note that if we would 'reverse' the problem, and ask the question: If $y$ increases by 1, what is the expected increase in $x$? We can compute the answer as follows:

$$\frac{cov(x,y)}{var(y)}$$

Answer 2

In simple linear regression we are dealing with the model \begin{eqnarray*} y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \mbox{for} \quad i=1,\cdots,n. \end{eqnarray*}

Let $\boldsymbol{y} = \left(y_1, \cdots, y_n\right)^{\prime}$, $\boldsymbol{x} = \left(x_1, \cdots, x_n\right)^{\prime}$, and $\boldsymbol{\beta} = \left(\beta_0, \beta_1\right)$. Moreover, the design matrix is $\boldsymbol{X} = \left[\boldsymbol{1}_n, \boldsymbol{x} \right]$. It may be shown that the MLE of $\boldsymbol{\beta}$ is \begin{eqnarray*} \widehat{\boldsymbol{\beta}} &=& \left(\boldsymbol{X}^{\prime}\boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \\ &=& \frac{1}{n\sum_{i=1}^n x_i^2 - n^2 \bar{x}^2} \begin{pmatrix} \sum_{i=1}^n x_i^2 & -n\bar{x} \\ -n\bar{x} & n \end{pmatrix} \begin{pmatrix} n \bar{y} \\ \sum_{i=1}^n x_iy_i \end{pmatrix}. \end{eqnarray*} Clearly, the second term, $\widehat{\beta}_1$, is given by \begin{eqnarray*} \widehat{\beta}_1 &=& \frac{n\sum_{i=1}^n x_iy_i - n^2 \bar{x}\bar{y}}{n\sum_{i=1}^n x_i^2 - n^2 \bar{x}^2} \\ &=& \frac{\frac{\sum_{i=1}^n x_iy_i - n \bar{x}\bar{y}}{n-1}}{\frac{\sum_{i=1}^n x_i^2 - n \bar{x}^2}{n-1}} \\ &=& \frac{cov(x,y)}{var(x)}, \end{eqnarray*} where $var$ and $cov$ represent their sample counterparts.

Answer 3

The first thing to mention is that the $\frac{cov(x,y)}{var(x)}$ will only be the $b_1$ in the case of this regression: $y = b_0 + b_1x$, that is, when we only have two variables. If we have a third, for example, this breaks down as you can test with the R code below:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)
cov(X,Y)/var(X)

Running the code above you will get the same value from the last two lines, which is $0.9882329$. But if you run the code below:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
Z <- X + rnorm(N)
lm(Y~X + Z)
cov(X,Y)/var(X)

You will get $0.9887337$ for $b_1$ and $0.9882329$ for $\frac{cov(X,Y)}{var(X)}$. And even though these values are close here, this is not always the case. If I had Z <- Y + rnorm(N), you would see $0.47370$ for $b_1$.

Now for your question. The variance measures how spread are the data points of a variable when compared to its mean. The covariance, in a way, measures if the spread in variable X follows the spread in variable Y. See the example below.

$cov(X,Y) = E[(X-E[X])(Y-E[Y])]$

Let's look at a data point at a time. We estimated that the mean for $X$ is $5$ and for $Y$ it is $10$. The first value we have for $X$ in the dataset is $20$, which is above its mean ($20\gt 5$). The first value we have for $Y$ is $0$, which is below its mean ($0 \lt 10$). Let's plug these values in the equation for covariance.

$(X-E[X])(Y-E[Y]) = (20-5)(0-10) = (15)(-10) = -150$.

Another data point now. $6$ for $X$ and $5$ for $Y$. Again, we will get a negative value.

$(X-E[X])(Y-E[Y]) = (6-5)(5-10) = (1)(-5) = -5$

If we keep doing this, and keep seeing this behaviour, it's easy to see that at the end we will have a negative covariance. $X$ values will to some extent be frequently above its mean, while $Y$ values will to some extent be frequently below its mean. Knowing that the Pearson correlation coefficient (PCC) is the standardised covariance ($ r = \frac{cov(X,Y)}{\sigma X \sigma Y}$), we would then equally get a negative value, that is, a negative correlation.

What does a negative correlation tell you right away? If I don't tell you the value of the PCC, the only thing the sign gives you is the "orientation" of the slope (increasing or decreasing). What's the $b_1$ coefficient in $y = b_0 + b_1x$? The slope! :-)

Based on that, I assume it's clear that the covariance of $X$ and $Y$ is proportional to the slope ($b_1$) of the regression line of $Y$, as dependent variable, and $X$ as independent variable.

However, there is a catch here. If you try to do a regression swapping who is the independent variable and who is the dependent one, you will realize the coefficients change! See the code below.

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)$coefficients
lm(X~Y)$coefficients

$b_1$ for the first regression will be $0.98823293$, as we already saw, but for the second regression we will get $0.50075232$. Based on what I said about $cov(X,Y)$, someone may think of trying $cov(Y,X)$ but covariance is symmetric, that is, $cov(X,Y) = cov(Y,X)$. If you divide this by the variance of the independent variable, you will get the right value. The code below shows this:

set.seed(2021)
N <- 1000
X <- rnorm(N)
Y <- X + rnorm(N)
lm(Y~X)$coefficients
lm(X~Y)$coefficients
cov(Y,X)
cov(X,Y)

cov(X,Y)/var(X)
cov(X,Y)/var(Y)

For the derivation of the least squares estimators of the slope and intercept in linear regression, this is a complete and slow-paced video.