Valerio Filoso (2013) writes:

Most econometrics textbooks limit themselves to providing the formula for the $\beta$ vector of the type

$$\beta = (X′X)^{-1} X'Y.$$

Although compact and easy to remember, this formulation is a sort black box, since it hardly reveals anything about what really happens during the estimation of a multivariate OLS model. Furthermore, the link between the $$\beta$$ and the moments of the data distribution disappear buried in the intricacies of matrix algebra. Luckily, an enlightening interpretation of the $$\beta$$s in the multivariate case exists and has relevant interpreting power. It was originally formulated more than seventy years ago by Frisch and Waugh (1933), revived by Lovell (1963), and recently brought to a new life by Angrist and Pischke (2009) under the catchy phrase regression anatomy. According to this result, given a model with K independent variables, the coefficient $\beta$ for the k-th variable can be written as

$$ \beta_k = \frac{cov(y,\tilde{x}_k)}{var(\tilde x)}$$

where $\tilde x_k$ is the residual obtained by regressing $x_k$ on all remaining $K − 1$ independent variables.

The result is striking since it establishes the possibility of breaking a multivariate model with K independent variables into $K$ bivariate models and also sheds light into the machinery of multivariate OLS. This property of OLS does not depend on the underlying Data Generating Process or on its causal interpretation: it is a mechanical property of the estimator which holds because of the algebra behind it.

Judd et al. (2017) have a nice explanation of this too, pp.107-116.

I'm not sure how you would break a multivariate model with $K$ independent variables into $K$ bivariate models, it seems like you would need more. For example with three independent variables and a constant ($K=4$), I worked out you need 5 base-level regressions (if you don't count the redundant ones in light blue) and 11 regressions of residuals on residuals. Am I missing something?

enter image description here

  • $\begingroup$ Your formula for $\beta_k$ is explicit: that is an estimate in a bivariate model (based on $(\tilde x_k,y)$). Maybe what bothers you is that the quotation hasn't included in its count all the other models needed to regress each regressor on all the remaining ones. $\endgroup$ – whuber Jan 10 '18 at 22:20
  • $\begingroup$ oh ok! so the quote only refers to the second box on the top left of my diagram, in which there are indeed K=3 regressions. If we want to start from only bivariate regressions of the original regressors (as opposed to residuals), we need far more than K of them. Yes? $\endgroup$ – tmksitt Jan 10 '18 at 23:41
  • 1
    $\begingroup$ Yes. The total number of bivariate regressions needed grows quadratically with $K$. (If it grew only linearly, we would have discovered a dramatic new improvement in the time needed to solve large multiple regression problems!) $\endgroup$ – whuber Jan 10 '18 at 23:43
  • $\begingroup$ I wrote it up here: thomas-sittler.github.io/reganat $\endgroup$ – tmksitt Jan 11 '18 at 20:59