When we do multiple regressions and say we are looking at the average change in the $y$ variable for a change in an $x$ variable, holding all other variables constant, what values are we holding the other variables constant at? Their mean? Zero? Any value?
I'm inclined to think it's at any value; just looking for clarification. If anyone had a proof, that would be great too.
You are right. Technically, it is any value. However, when I teach this I usually tell people that you are getting the effect of a one unit change in $X_j$ when all other variables are held at their respective means. I believe this is a common way to explain it that is not specific to me.
I usually go on to mention that if you don't have any interactions, $\beta_j$ will be the effect of a one unit change in $X_j$, no matter what the values of your other variables are. But I like to start with the mean formulation. The reason is that there are two effects of including multiple variables in a regression model. First, you get the effect of $X_j$ controlling for the other variables (see my answer here). The second is that the presence of the other variables (typically) reduces the residual variance of the model, making your variables (including $X_j$) 'more significant'. It is hard for people to understand how this works if the other variables have values that are all over the place. That seems like it would increase the variability somehow. If you think of adjusting each data point up or down for the value of each other variable until all the rest of the $X$ variables have been moved to their respective means, it is easier to see that the residual variability has been reduced.
I don't get to interactions until a class or two after I've introduced the basics of multiple regression. However, when I do get to them, I return to this material. The above applies when there are not interactions. When there are interactions, it is more complicated. In that case, the interacting variable[s] is being held constant (very specifically) at $0$, and at no other value.
If you want to see how this plays out algebraically, it is rather straight-forward. We can start with the no-interaction case. Let's determine the change in $\hat Y$ when all other variables are held constant at their respective means. Without loss of generality, let's say that there are three $X$ variables and we are interested in understanding how the change in $\hat Y$ is associated with a one unit change in $X_3$, holding $X_1$ and $X_2$ constant at their respective means:
\begin{align} \hat Y_i &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3X_{3i} \\ \hat Y_{i'} &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) \\ ~ \\ &\text{subtracting the first equation from the second:} \\ ~ \\ \hat Y_{i'} - \hat Y_i &= \hat\beta_0 - \hat\beta_0 + \hat\beta_1\bar X_1 - \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 - \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) - \hat\beta_3X_{3i} \\ \Delta Y &= \hat\beta_3X_{3i} + \hat\beta_3 - \hat\beta_3X_{3i} \\ \Delta Y &= \hat\beta_3 \end{align}
Now it is obvious that we could have put any value in for $X_1$ and $X_2$ in the first two equations, so long as we put the same value for $X_1$ ($X_2$) in both of them. That is, so long as we are holding $X_1$ and $X_2$ constant.
On the other hand, it does not work out this way if you have an interaction. Here I show the case where there is an $X_1X_3$ interaction term:
\begin{align} \hat Y_i &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3X_{3i} \quad\quad\ \! + \hat\beta_4\bar X_1X_{3i} \\ \hat Y_{i'} &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) + \hat\beta_4\bar X_1(X_{3i}\!+\!1) \\ ~ \\ &\text{subtracting the first equation from the second:} \\ ~ \\ \hat Y_{i'} - \hat Y_i &= \hat\beta_0 - \hat\beta_0 + \hat\beta_1\bar X_1 - \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 - \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) - \hat\beta_3X_{3i} + \\ &\quad\ \hat\beta_4\bar X_1(X_{3i}\!+\!1) - \hat\beta_4\bar X_1X_{3i} \\ \Delta Y &= \hat\beta_3X_{3i} + \hat\beta_3 - \hat\beta_3X_{3i} + \hat\beta_4\bar X_1 X_{3i} + \hat\beta_4\bar X_1 - \hat\beta_4\bar X_1X_{3i} \\ \Delta Y &= \hat\beta_3 + \hat\beta_4\bar X_1 \end{align}
In this case, it is not possible to hold all else constant. Because the interaction term is a function of $X_1$ and $X_3$, it is not possible to change $X_3$ without the interaction term changing as well. Thus, $\hat\beta_3$ equals the change in $\hat Y$ associated with a one unit change in $X_3$ only when the interacting variable ($X_1$) is held at $0$ instead of $\bar X_1$ (or any other value but $0$), in which case the last term in the bottom equation drops out.
In this discussion, I have focused on interactions, but more generally, the issue is when there is any variable that is a function of another such that it is not possible to change the value of the first without changing the respective value of the other variable. In such cases, the meaning of $\hat\beta_j$ becomes more complicated. For example, if you had a model with $X_j$ and $X_j^2$, then $\hat\beta_j$ is the derivative $\frac{dY}{dX_j}$ holding all else equal, and holding $X_j=0$ (see my answer here). Other, still more complicated formulations are possible as well.
The math is simple, just take the difference between 2 models with one of the x variables changed by 1 and you will see that it does not matter what the other variables are (given there are no interactions, polynomial, or other complicating terms).
One example:
$y_{[1]} = b_0 + b_1 \times x_1 + b_2 \times x_2$
$y_{[2]} = b_0 + b_1 \times (x_1 + 1) + b_2 \times x_2$
$y_{[2]} - y_{[1]} = b_0 - b_0 + b_1\times x_1 - b_1\times x_1 + b_1 \times 1 + b_2 \times x_2 - b_2 \times x_2 = b_1$
I believe you are referring to dependence in covariates ($X_i$). So if the model is $$Y=\beta_{0}+\beta_{1}X_1+\beta_{2}X_2$$ the effect of $X_i$ on $Y$ all other things being equal would be $\frac{\Delta{Y}}{\Delta{X_i}}$ for any $\Delta{X_i}$ with all other $X_j$ held constant at any value.
Keep in mind that is possible that $X_1$ and $X_2$ are dependent (e.g. functions of each other) without necessarily showing a significant interaction in the linear model ($\beta_{12}=0$ in $Y=\beta_{0}+\beta_{1}X_1+\beta_{2}X_2+\beta_{12}X_1X_2$).
Just as an interesting tangent here is an example: Let $X_1\sim N(0,\sigma_1^2)$ and $X_2=X_1^{2}+N(0,\sigma_2^2)$ then clearly any change in $X_1$ will affect $X_2$. However the covariance between the two is zero. $$cov(X_1,X_2)=E(X_1X_2)-E(X_1)E(X_2)$$ $$=E[X_1(X_1^2+a)]-E(X_1).E(X_1^2-a)\,with\,a\sim N(0,\sigma_2^2)$$ $$=E(X_1^3)-E(X_1.a)-0.E(X_1^2-a)=0-0-0=0$$
So in reality a change in $X_1$ would be associated with a change in $X_2$ and that $\frac{\Delta{Y}}{\Delta{X_i}}$ would not cover what really would occur if you alter $X_1$. But $\frac{\Delta{Y}}{\Delta{X_i}}$ would still be described as the effect of $X_i$ on $Y$ all things being equal.
This is comparable to the difference between a full derivative and a partial derivate (the analog of $\frac{\Delta{Y}}{\Delta{X_i}}$) in a differential equation.
