6
$\begingroup$

One formula for the least squares linear regression line with independent variable $X$ and response variable $Y$ is:

$\hat{\beta} = \frac{Cov(X, Y)}{Var(X)}$

This formula screams deeper intuition but I can't quite find it. Is there some intuition that makes this formula "obvious"?

$\endgroup$

2 Answers 2

2
$\begingroup$

Define "obvious"! What's obvious to you doesn't need be obvious to someone else. So I can only offer my perspective.

First, I hope it's obvious that the slope $\beta$ of the line doesn't change if we shift the data around; only the intercept changes. So, to simplify the formulas, we can, without loss of generality, assume that our data are centred around the origin: $\bar x = \bar y = 0$. In that case the covariance-based formula simplifies to

\begin{align} \beta &= \frac{Cov(X, Y)}{Var(X)} \\ &= \frac{\sum_i (x_i - \bar x) (y_i - \bar y)}{\sum_i (x_i - \bar x)^2} \\ &= \frac{\sum_i x_i y_i}{\sum_i x_i^2} \\ \end{align}

This is, incidentally, the same formula we obtain by minimising least squares. From the above assumption of the centred data it follows that the intercept is zero, so it suffices to solve: $$ \min_{\beta} \frac{1}{2} \sum_i (y_i - \beta x_i)^2 $$

We find the minimum by setting the first derivative to zero: $$ \sum_i (y_i - \beta x_i)x_i = 0 $$

Solving for $\beta$ produces again: $$ \beta = \frac{\sum_i x_i y_i}{\sum_i x_i^2} $$

Now, agreed, this is again mathematical formalism. Another, more "intuitive" way of looking at it is to observe that each $x_i y_i$ is the area of a rectangle with sides $x_i$ and $y_i$. Equally, each $x_i^2$ is the area of a square with sides $x_i$. So the sum in the numerator above can be interpreted as the average rectangle area, and in the denominator as the average square (both scaled by the number of points, $N$, but these cancel out, so we can ignore them). The slope $\beta$ is then the ratio of these two areas. Now, if we construct the rectangle and the square to have the same base, $\tilde x$:

$$ \tilde x = \sqrt{\sum_i x_i^2} $$

then the other side of the "average rectangle" is given by

$$ \tilde y = \frac{\sum_i x_i y_i}{\tilde x} $$

It is then straightforward to see that

$$ \beta = \frac{\tilde y}{\tilde x} $$

or, graphically,

graphical beta

$\endgroup$
3
  • $\begingroup$ From your geometric intuition explanation above, it seems unclear why the absolute area of orange rectangle must be necessarily less than the green one? $\endgroup$
    – cinch
    Commented Dec 23, 2022 at 20:08
  • $\begingroup$ @mohottnad It doesn't. Why should it? $\endgroup$
    – Igor F.
    Commented Dec 23, 2022 at 20:26
  • $\begingroup$ I mean in the standardized variables case we know the absolute area of orange rectangle must be necessarily less than the green one, but above line of geometric explanation seems missing a possible mechanism to enforce this regularization as you confirmed above. $\endgroup$
    – cinch
    Commented Dec 23, 2022 at 23:30
0
$\begingroup$

Answer 1 (simple): The simple answer is that $\beta$ is the sensitivity of the response $y$ with respect to the regressor $X$ (assuming the relationship is true). Its estimator $\hat \beta$, which is what you quoted, is the result (the slope) of "forcing" (i.e. fitting) a line (consisting of an intercept and slope) to go through all the points, under specific conditions (i.e. minimizing squared distance). If $\beta=1$ then $y$ moves at the same rate than $X$, i.e. if $X$ moves up/down 1 unit then $y$ also moves up/down 1 unit on average. More/less than 1 unit if $\beta \gt 1 / \beta<1$. Since the formula for $\hat \beta$ is only a special case with 1 single regressor and it also is the result of an optimization problem we could simply look at the formula as such, i.e. it's "just coincidence" that the covariance and variance come out.

Answer 2 (advanced): But I think you are looking for something deeper than the above. In financial mathematics (more specifically: portfolio theory) there's a very interesting interpretation to precisely this formula (for the OLS estimator of the slope with 1 single regressor), which - even if maybe over-the-top - could be generalized beyond this field. It interprets the covariance as being the systemic risk and the variance as the non-systemic (or idiosyncratic) risk. Let me explain:

In financial mathematics the CAPM (capital asset pricing model) is a formula for calculating the expected return of a financial instrument (e.g. share). According to financial theory (see Markowitz portfolio theory) the average return of an individual stock can be explained by the average return of the market. Without going into too much detail, the individual excess stock return (the response $y=r_i-r_f$) is the excess return of the market (the regressor $X$) times the beta factor:

$$r_i = r_f + \beta \cdot\underbrace{(r_M-rf)}_{\textrm{excess return market}}$$

where $r_f$ is the risk-free rate (can be ignored at this point). The beta factor $\beta$ is defined as

$$ \beta = \frac{Cov(r_i,r_M)}{Var(r_M)} $$

and it is interpreted as follows: if the investor that is interested in calculating the return $r_i$ for stock $i$ is well-diversified, then they won't care about the individual risk of the stock, i.e. risk inherent to that specific company beyond the current state of the economy, here defined as the variance, since that can be diversified away. Instead, what they will be interested in is only the systematic risk of the stock, i.e. the risk that the company has due to the underlying state of the economy.

To make it clear what is meant by "diversifying away": imagine we have 1 single stock, represented by a single random variable $X_1$. Obviously, the "risk" we have will be the whole variance $\sigma_1^2$ (assuming we regard both up and down swings as risk). But what if we add another stock, represented by the random variable $X_2$. Then the variance of our portfolio $0.5*X_1+0.5*X_2$ (i.e. sum of equal proportions of all random variables) will now be $0.5^2\sigma_1^2+0.5^2\sigma_2^2+0.5^2\sigma_{1,2}$ where $\sigma_{1,2}$ is the covariance between them. If we assume we keep adding random variables $X_3, X_4, \dots X_\infty$ with the same variance and covariances to our portfolio, something interesting happens: we will converge towards the covariance (assuming the covariance is smaller than the variance). Here is a simple simulation to show what I mean:

sim_cov_portfolio

The now infinite (or very large) portfolio is the market (i.e. all stocks weighted equally and summed up). Obviously, the variance is now as small as can be and is equal to the individual covariance. Hence, if we add another stock to it, given that we already have this whole portfolio, the variance of our portfolio will only grow by the additional covariance of the stock with the market (i.e. with all other stocks), not by its individual variance. Hence, we are only interested in the covariance. So, the beta-factor is the ratio of the newly added risk to the portfolio (cov of stock to market) to the risk of the portfolio (var of market).

Interestingly enough in textbooks the beta-factor is initially defined as the formula above and only thereafter it is mentioned that it can be estimated via regression, so pretty much the inverse of the logic of the first answer.

(An interesting additional info: if the assumption of the fully diversified investor is not made (as mentioned above), the formula for the beta-factor changes to $\beta = Var(r_i)/Var(r_M)$, i.e. to include the idiosyncratic risk of the stock as well. This is called the total beta.)

Conclusion (TL;DR): Now you could try and generalize this: $\hat \beta$ is the ratio of the variance that $X$ adds to $y$, if we interpret $y$ as a portfolio of very many $X$s, to the the variance of $y$, i.e. it is the proportion/factor of the "systemic" risk that $X$ brings to $y$ to the "systemic" risk as whole (where the "system" is $y$). (Mind you, take this with a pinch of salt, I haven't given it much more thought. I personally stick with Answer 1 since it's generalizabe beyond 1 regressor, when the formula for the OLSE changes.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.