Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

When we do multiple regressions and say we are looking at the average change in the $y$ variable for a change in an $x$ variable, holding all other variables constant, what values are we holding the other variables constant at? Their mean? Zero? Any value?

I'm inclined to think it's at any value; just looking for clarification. If anyone had a proof, that would be great too.

share|improve this question
2  
I found example 10 in Peter Kennedy's paper very helpful in understanding this. –  Dimitriy V. Masterov Feb 3 at 21:10
    
Yeah, the bit about increasing the number of rooms while keeping square feet constant is a really observant point. That paper is actually a goldmine of useful ideas, it's going in the PhD notes. –  EconStats Feb 3 at 23:48

3 Answers 3

up vote 15 down vote accepted

You are right. Technically, it is any value. However, when I teach this I usually tell people that you are getting the effect of a one unit change in $X_j$ when all other variables are held at their respective means. I believe this is a common way to explain it that is not specific to me.

I usually go on to mention that if you don't have any interactions, $\beta_j$ will be the effect of a one unit change in $X_j$, no matter what the values of your other variables are. But I like to start with the mean formulation. The reason is that there are two effects of including multiple variables in a regression model. First, you get the effect of $X_j$ controlling for the other variables (see my answer here). The second is that the presence of the other variables (typically) reduces the residual variance of the model, making your variables (including $X_j$) 'more significant'. It is hard for people to understand how this works if the other variables have values that are all over the place. That seems like it would increase the variability somehow. If you think of adjusting each data point up or down for the value of each other variable until all the rest of the $X$ variables have been moved to their respective means, it is easier to see that the residual variability has been reduced.

I don't get to interactions until a class or two after I've introduced the basics of multiple regression. However, when I do get to them, I return to this material. The above applies when there are not interactions. When there are interactions, it is more complicated. In that case, the interacting variable[s] is being held constant (very specifically) at $0$, and at no other value.

If you want to see how this plays out algebraically, it is rather straight-forward. We can start with the no-interaction case. Let's determine the change in $\hat Y$ when all other variables are held constant at their respective means. Without loss of generality, let's say that there are three $X$ variables and we are interested in understanding how the change in $\hat Y$ is associated with a one unit change in $X_3$, holding $X_1$ and $X_2$ constant at their respective means:

\begin{align} \hat Y_i &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3X_{3i} \\ \hat Y_{i'} &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) \\ ~ \\ &\text{subtracting the first equation from the second:} \\ ~ \\ \hat Y_{i'} - \hat Y_i &= \hat\beta_0 - \hat\beta_0 + \hat\beta_1\bar X_1 - \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 - \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) - \hat\beta_3X_{3i} \\ \Delta Y &= \hat\beta_3X_{3i} + \hat\beta_3 - \hat\beta_3X_{3i} \\ \Delta Y &= \hat\beta_3 \end{align}

Now it is obvious that we could have put any value in for $X_1$ and $X_2$ in the first two equations, so long as we put the same value for $X_1$ ($X_2$) in both of them. That is, so long as we are holding $X_1$ and $X_2$ constant.

On the other hand, it does not work out this way if you have an interaction. Here I show the case where there is an $X_1X_3$ interaction term:

\begin{align} \hat Y_i &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3X_{3i} \quad\quad\ \! + \hat\beta_4\bar X_1X_{3i} \\ \hat Y_{i'} &= \hat\beta_0 + \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) + \hat\beta_4\bar X_1(X_{3i}\!+\!1) \\ ~ \\ &\text{subtracting the first equation from the second:} \\ ~ \\ \hat Y_{i'} - \hat Y_i &= \hat\beta_0 - \hat\beta_0 + \hat\beta_1\bar X_1 - \hat\beta_1\bar X_1 + \hat\beta_2\bar X_2 - \hat\beta_2\bar X_2 + \hat\beta_3(X_{3i}\!+\!1) - \hat\beta_3X_{3i} \\ &\ \ + \hat\beta_4\bar X_1(X_{3i}\!+\!1) - \hat\beta_4\bar X_1X_{3i} \\ \Delta Y &= \hat\beta_3X_{3i} + \hat\beta_3 - \hat\beta_3X_{3i} + \hat\beta_4\bar X_1 X_{3i} + \hat\beta_4\bar X_1 - \hat\beta_4\bar X_1X_{3i} \\ \Delta Y &= \hat\beta_3 + \hat\beta_4\bar X_1 \end{align}

In this case, it is not possible to hold all else constant. Because the interaction term is a function of $X_1$ and $X_3$, it is not possible to change $X_3$ without the interaction term changing as well. Thus, $\hat\beta_3$ equals the change in $\hat Y$ associated with a one unit change in $X_3$ only when the interacting variable ($X_1$) is held at $0$ instead of $\bar X_1$ (or any other value but $0$), in which case the last term in the bottom equation drops out.

In this discussion, I have focused on interactions, but more generally, the issue is when there is any variable that is a function of another such that it is not possible to change the value of the first without changing the respective value of the other variable. In such cases, the meaning of $\hat\beta_j$ becomes more complicated. For example, if you had a model with $X_j$ and $X_j^2$, then $\hat\beta_j$ is the derivative $\frac{dY}{dX_j}$ holding all else equal, and holding $X_j=0$ (see my answer here). Other, still more complicated formulations are possible as well.

share|improve this answer
    
Thanks gung, this answer is great on a couple of levels. Firstly it answers the main point I was interested in. Secondly, you predicted what my follow up question would be, because I was going to ask how this changed with the introduction of interaction terms. Thanks for the math as well. I know this question is kind of basic but I feel that you can never be too explicit with these concepts. –  EconStats Feb 3 at 23:36
    
You're welcome, @EconStats. There is no problem with including the math, sometimes it makes it much easier to understand what is going on. –  gung Feb 3 at 23:54
    
Well I have to say that when you subtracted the first equation from the second equation it finally confirmed my original thoughts that it doesn't matter what the values of $X_2$ and $X_3$ are, as long as the are the same in both equations. It seems so obvious to me know but I had never thought about calculating the $\beta$ that way before. Definite light bulb moment for me. –  EconStats Feb 4 at 0:19
    
You can also take the derivative of $Y$ wrt $X_j$ and it will get you to the same place, but this is easier math (essentially high-school algebra), so it will be accessible to a broader audience. –  gung Feb 4 at 1:02

The math is simple, just take the difference between 2 models with one of the x variables changed by 1 and you will see that it does not matter what the other variables are (given there are no interactions, polynomial, or other complicating terms).

One example:

$y_{[1]} = b_0 + b_1 \times x_1 + b_2 \times x_2$

$y_{[2]} = b_0 + b_1 \times (x_1 + 1) + b_2 \times x_2$

$y_{[2]} - y_{[1]} = b_0 - b_0 + b_1\times x_1 - b_1\times x_1 + b_1 \times 1 + b_2 \times x_2 - b_2 \times x_2 = b_1$

share|improve this answer

I believe you are referring to dependence in covariates ($X_i$). So if the model is $$Y=\beta_{0}+\beta_{1}X_1+\beta_{2}X_2$$ the effect of $X_i$ on $Y$ all other things being equal would be $\frac{\Delta{Y}}{\Delta{X_i}}$ for any $\Delta{X_i}$ with all other $X_j$ held constant at any value.

Keep in mind that is possible that $X_1$ and $X_2$ are dependent (e.g. functions of each other) without necessarily showing a significant interaction in the linear model ($\beta_{12}=0$ in $Y=\beta_{0}+\beta_{1}X_1+\beta_{2}X_2+\beta_{12}X_1X_2$).

Just as an interesting tangent here is an example: Let $X_1\sim N(0,\sigma_1^2)$ and $X_2=X_1^{2}+N(0,\sigma_2^2)$ then clearly any change in $X_1$ will affect $X_2$. However the covariance between the two is zero. $$cov(X_1,X_2)=E(X_1X_2)-E(X_1)E(X_2)$$ $$=E[X_1(X_1^2+a)]-E(X_1).E(X_1^2-a)\,with\,a\sim N(0,\sigma_2^2)$$ $$=E(X_1^3)-E(X_1.a)-0.E(X_1^2-a)=0-0-0=0$$

So in reality a change in $X_1$ would be associated with a change in $X_2$ and that $\frac{\Delta{Y}}{\Delta{X_i}}$ would not cover what really would occur if you alter $X_1$. But $\frac{\Delta{Y}}{\Delta{X_i}}$ would still be described as the effect of $X_i$ on $Y$ all things being equal.

This is comparable to the difference between a full derivative and a partial derivate (the analog of $\frac{\Delta{Y}}{\Delta{X_i}}$) in a differential equation.

share|improve this answer
    
Can you clarify the part about $X_1$ and $X_2$ being "not independent"? Eg, variables can be not independent in the sense of being collinear w/o causing any problems relative to this issue. OTOH, there can be a real interaction b/t 2 variables & have those 2 variables be completely independent. –  gung Feb 3 at 22:04
    
I meant functionally dependent but possibly with correlation zero (so expected interaction effect is zero) e.g. $X_2=X_1^2+N(0,1)$ with $X_1$ and $X_2$ both $N(0,\sigma^2)$ –  Hans Roggeman Feb 3 at 22:52
    
I suspected you did ;-), but could you perhaps edit your answer so that someone who doesn't just recognize what you meant wouldn't be confused? –  gung Feb 3 at 22:56
    
Will do :) - thanks for help. –  Hans Roggeman Feb 3 at 23:00
    
Thanks Hans, I was actually trying to get at the point that gung made but this is a good example for when the two variables are dependent. –  EconStats Feb 4 at 0:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.