Interpretation of OLS regression coefficients Much of the empirical literature seems to interpret the coefficient b of a simple linear (OLS) regression Y = a + bX + e as follows: a one-unit increase in X will, ON AVERAGE, cause a b unit increase in Y. My question is - where is the "on average" coming from? Is this interpretion ok? (although we use LS criterion)
 A: This is a really interesting question---especially in the context of multiple regression. 

Last year, I was the teaching assistant for an intro statistics (for business majors) course where the instructor was John Tukey's student. The instructor pushed for the interpretation:

On average, after removing the affects of the other covariates, the response changes by $b$ [y units] per [x units].

The "on average" is appearing since the increment by $b$ is not deterministic--the increment is corrupted by mean zero noise.
I like this framing because it (a) keeps the interpretation sounding clearly non-casual and (b) avoids the colinearity problems in the standard interpretation posed in the original post. Indeed, when we change $x_1$ by one unit, if $x_1$ and $x_2$ are correlated, then $x_2$ will, on average, change too!
The above interpretation relies on a property of partial regression plots, which are nicely explained here: What does an Added Variable Plot (Partial Regression Plot) explain in a multiple regression?
Using the notation from this question without explanation, the reason is just that the regression of $(I-H_{-j})y$ against $(I-H_{-j})x_j$ has slope \begin{align*}
 [x_{j}^T (I - H_{-j})x_j]^{-1}[x_j^T (I - H_{-j})y]
 & = [x_{j}^T (I - H_{-j})x_j]^{-1}[x_j^T (I - H_{-j})(x_j \hat\beta_j + r)] \\
 & = \hat\beta_j + [x_{j}^T (I - H_{-j})x_j]^{-1}[x_j^T (I - H_{-j})r] \\
 & = \hat\beta_j,
\end{align*}
since $(I-H_{-j})x_j \in \mathrm{im}(X)$ and $r \perp \mathrm{im}(X)$.

I think we can get even more interesting, though. When $y = f(X) + \epsilon$, and $f(X) \neq X \beta^*$ isn't linear, there still exists interesting explanations of what the OLS estimates for the slope produce. This is discussed excellently here:

Buja, A., Berk, R., Brown, L., George, E., Pitkin, E., Traskin, M., and Zhao, L. (2014). Models as Approximations, Part I: A Conspiracy of Nonlinearity and Random Regressors in Linear Regression. 

In that paper, it'd discussed that OLS is, in this non linear case, estimating the best linear approximation, and interpretation with similar spirit to above are given.
