# Understanding the meaning of the parameters in the linear regression model

When I first time learn multiple linear regression, I remember the interpretation of the regression coefficient is that: the marginal contribution of a specific predictor.

Now I am rethinking this interpretation and create an example to ask myself a question: is this interpretation always correct?

Suppose I have 2 dummy variables in my data set $X_i$ and $Z_i$, to make it concrete, they are defined as the following \begin{equation*} X_i=\begin{cases} &1,~\text{if people $i$ go to school before}\\ &0,~\text{otherwise} \end{cases},~~~Z_i=\begin{cases} &1,~\text{if people $i$ drink milk}\\ &0,~\text{otherwise} \end{cases} \end{equation*} Let $Y_i$ be the response variable, say the wage of people $i$. There are 4 combinations of the value of $(X_i,Z_i)$. If we can FORCE all people in the population have a specific value of $(X_i,Z_i)$, then we will be able to get a wage distribution, denoted by random variable $Y^Z_{Xi}$. So we have $Y^1_{1i}$, $Y^1_{0i}$, $Y^0_{0i}$ and $Y^0_{1i}$. Note that, without loss of generality, we can always assume that \begin{equation*} Y^1_{1i}=\mu^1_1+\epsilon^1_{1i},~Y^1_{0i}=\mu^1_0+\epsilon^1_{0i},~Y^0_{1i}=\mu^0_1+\epsilon^0_{1i},~Y^0_{0i}=\mu^0_0+\epsilon^0_{0i}\\ \end{equation*} where $\mu^X_Z$ is the mean of the wage if the people in the population ALL have value $X$ and $Z$. After rearrangement, it is easily to write $Y_i$, which is the observed wage in the real world, as \begin{align*} Y_i&=\mu^0_0+X_i(\mu^0_1-\mu^0_0)+Z_i(\mu^1_0-\mu^0_0)+X_iZ_i(\mu^1_1-\mu^0_1-\mu^1_0+\mu^0_0)\\ &+\underset{\epsilon_i}{\underbrace{\epsilon^0_{0i}+X_i(\epsilon^0_{1i}-\epsilon^0_{0i})+Z_i(\epsilon^1_{0i}-\epsilon^0_{0i})+X_iZ_i(\epsilon^1_{1i}-\epsilon^0_{1i}-\epsilon^1_{0i}+\epsilon^0_{0i})}} \end{align*} To write it clearly, we have $$Y_i=a+bX_i+cZ_i+dX_iZ_i+\epsilon_i,\label{eq:2D}$$ where $a=\mu^0_0, b=\mu^0_1-\mu^0_0, c=\mu^1_0-\mu^0_0, d=\mu^1_1-\mu^0_1-\mu^1_0+\mu^0_0$ Note that this equation decribes the TRUTH of the world (without making ANY assumption) and all of these coefficients have real meaning, e.g., $b$ is the increment of mean wage if the population have no milk but change from no school to all go to school, i.e., the effect of schooling when $Z_i=0$ (no milk). Note that whether we can consistently estimate these coefficient is another story.

Now, what if when you see the data set $\{(Y_i,X_i,Z_i)\}_{i=1}^n$, you build up the following model $$Y_i=a'+b'X_i+c'Z_i+\epsilon'_i,$$ i.e., you didn't incorporate the term $X_iZ_i$ in your model, which means that you dump this term into the error term, then here comes my questions:

(1) if you don't include the term $X_iZ_i$, is it always true that the error term will be correlated with the predictors? Intuitively, it looks yes, since how could the random variable $X$ and $XZ$ being uncorrelated? Then this could implies that OLS cannot consistently estimate $a',b',c'$. If for some reason that I don't know, $\epsilon'_i$ is uncorrelated with $X$ and $Z$, then we know that the omitted variable $XZ$ does not affect the OLS, so in this case, $a'=a,b'=b,c'=c$, but then what is the correct interpretation of these coefficients? $b'$ is still: "the increment of mean wage if the population have no milk but change from no school to all go to school"?

(2) somewhat related to (1), if initially I only have data $\{(Y_i,X_i)\}_{i=1}^n$, then I build the model $Y_i=a'' + b'' X_i+\epsilon''_i$, where $a''=\mathbb{E}[Y_{0i}]$, i.e., the mean wage of population if nobody go to school regardless of whether they drink milk and $b''=\mathbb{E}[Y_{1i}-Y_{0i}]$. Now somehow I observe the information about whether each individual drink milk or not. Then I project the error term $\epsilon''_i$ onto $Z_i$, i.e., $\epsilon''_i=c'' Z_i+\nu_i$. Then I run $Y_i=a'' + b'' X_i + c''_i Z_i + \nu_i$, what is the meaning of $a''_i$ and $b''_i$ in the new model? Are they the same as in the old model (both in the theoretical value and the interpretation)?

(3) For any dataset we have (with a response variable $Y$ and a predictors vector $W$), we can always write $Y_i=\mathbb{E}[Y_i|W_i]+e_i$, where $e_i$ is the residual, which is defined as $e_i\triangleq Y_i-\mathbb{E}[Y_i|W_i]$. By definition, $e_i$ is uncorrelated with $W_i$. In this post, $W_i$ are all dummies, $W_i=(X_i,Z_i)$, then $\mathbb{E}[Y_i|W_i]$ is linear, i.e., you could write $\mathbb{E}[Y_i|W_i]$ as $\mathbb{E}[Y_i|X_i,Z_i]=\alpha+\beta X_i+\gamma Z_i+\delta X_iZ_i+e_i$. Note that those $\alpha,\beta,\gamma,\delta$ have nothing to do with the aforementioned $a,b,c,d$. Here those Greek letters have completely different interpretation, e.g., here $\beta$ means the impact of unit change in $X$ on $\mathbb{E}[Y_i|X_i,Z_i]$ when $Z=0$, which is a pure description about the relationship between the predictors and the conditional expectation, that has nothing to do with the counterfactual effect like the interpretation of $b$. So in this case, if we omit the variable $X_iZ_i$, and model the linear functional $\mathbb{E}[Y_i|X_i,Z_i]$ as $\mathbb{E}[Y_i|X_i,Z_i]=\alpha'+\beta'X_i+\gamma'Z_i+e'_i$, then $\beta'$ just means the impact of unit change in $X$ on $Y_i$ (no "when $Z=0$"). In empirical study, which interpretation of the parameters should we adopt? the $\beta$ one or the $b$ one? Note that as $e_i$ is the residual (NOT the $\epsilon_i$ aforementioned), which is defined as $e_i\triangleq Y_i-\mathbb{E}[Y_i|W_i]$, then by definition, $e_i$ is uncorrelated with $W_i$. Hence, in this case, the OLS is always consistent, in the sense that, OLS estimator will converge to those $\beta$'s (the Greek letter), which does not necessarily equal to $b$ (the English letter). I recall that when I learned econometrics, the consistency of OLS estimator is a big chunk of lectures, so I assume the correct interpretation of the coefficients in the linear regression model should be the $b$ one, rather than the $\beta$ one, since otherwise, the OLS estimator is always consistent. Did I miss something important?

To correct your understanding a bit, your interpretation of the model coefficients as the marginal contribution of a predictor is not one I've ever heard before. And I don't think it's right. Marginal suggests that it is not conditional. In an OLS model with two factors, $X$ and $Z$, the two respective coefficients $b$ and $c$ certainly differ from the results obtained by regressing $X$ and $Z$ separately and obtaining coefficients $b_m$ and $c_m$. In the two factor model, the interpretation of $b$ is "an expected mean difference in $Y$ comparing groups differing by one unit in $X$ having the same value of $Z$", whereas the marginal model would not have the "having the...". The idea of "forcing" people one unit higher is a causal interpretation, and with observational data and causal analysis, is a type of counterfactual reasoning, sort of rewinding time.

The model you've written is a heteroscedastic one, and the rearrangement would require terms of $(1-X)$ and $(1-Z)$ to obtain $\mu_{ij}$ for specific covariate values of $X_i Z_j$. So I'd recommend writing it like:

$$E[Y|X,Z] = \mu_{00}(1-X)(1-Z) + \mu_{01}(1-X)Z + \mu_{10}X(1-Z) + \mu_{11}XZ$$

Omitting the interaction term from this model produces fitted values which are a complex combination of the $\mu$s depending on the correlation of $X$ and $Z$, but they can be calculated by hand, and estimated consistently.

1. Yes it's true that omitting certain predictors from a model (what we call "model misspsecification") will result in errors which are correlated with the omitted predictors and the observed predictors if those two are correlated. This is something we really never observed because the error is a thing different from the residuals. If I fit the OLS model, the residuals will be orthogonal to any predictor in the model.

2. My first paragraph should address the difference in interpretation from the unadjusted model, and the adjusted one. The interpretation will not be the same. The actual value of the coefficient will not be the same unless $X$ and $Z$ are uncorrelated, but even when they are I would not interpret the regression models results as the same.

3. In your description of results you omitted an interaction term. In regression analysis we often consider exploring interaction terms as a type of post hoc analysis. What you might imagine is that when the interaction is present --suppose the object of inference is the relationship between $X$ and $Y$ controlling for $Z$ (binary)-- there is one slope comparing $X$ to $Y$ when $Z=0$ and another when $Z=1$, by omitting the interaction term, the estimated slope comparing $X$ to $Y$ is just a weighted average between the two $Z$ levels. In observational studies, what you are probably calling "empirical" studies", you mustn't use causal language in interpreting any of the findings, including words like "impact" "change" and "increase".

• In this case, i.e., 2 dummy predictors. I think the model $Y_i=a+bX_i+cZ_i+dX_iZ_i+\epsilon_i$ is the true model that generates the data. So the consistency of OLS estimator means the convergence to those $a,b,c,d$ right? Note that $a=\mu^0_0,b=\mu^0_1-\mu^0_0,c=\mu^1_0-\mu^0_0$ and $d=\mu^1_1-\mu^0_1-\mu^1_0+\mu^0_0$ regardless of the data is observational or experimental, then if $\epsilon$ is uncorrelated with predictors, will the interpretation of OLS estimator be different if the data is observational or experimental? Commented Aug 23, 2016 at 14:56
• If $\epsilon_i$ is correlated with the predictors, then OLS estimator won't converge to $(a,b,c,d)$, but will still converge, to something else which is $(\alpha,\beta,\gamma,\delta)$, that are the optimal solution to $\min~\mathbb{E}[(Y_1-\alpha -\beta X_i-\gamma Z_i-\delta X_iZ_i)^2]$, then how should we interpret those $\alpha,\beta,\gamma,\delta$? In this case, we say the OLS estimator is biased, which means $\beta\neq b$ right? Then can we say b is the trueexpected mean difference in Y comparing groups differing by one unit in X having the same value of Z" and $\beta$ is not the true... Commented Aug 23, 2016 at 15:03
• @KevinKim You are using correlation in the wrong sense. The first-order moment relating $\epsilon$ to the mean values $E[Y]$ will be 0 and it will converge to 0. You might need to make assumptions about the distribution of $X$ under repeated sampling. Estimating model under model misspecification will still provide consistent estimation. The "averaged out" version of the wrong model is still an estimable thing. Commented Aug 23, 2016 at 15:56
• @KevinKim In general, you should never interpret model effects as arising from a "true" model, I don't need to remind you: all models are wrong. Interpret the effects as being the "averaged first order trends". For instance, if $E[Y|X] = a X^2$ but I fit $E[Y|X] = a^{'} X$ the interpretation of $a'$ is the first order trend in these data. Use robust standard errors to obtain consistent error estimation in light of these issues. Commented Aug 23, 2016 at 15:59
• I used to believe "all models are wrong" until I come up with this example I made. I couldn't figure out why $Y_i=a+bX_i+cZ_i+dX_iZ_i+\epsilon_i$ is NOT the TRUE model, in my specific example. You can always write $Y^1_{1i}$ as a mean plus an error as I write in my OP and If the data represents the whole population of my study, then the mean always exist. So I think at least in this case, the true model exists and it makes sense to discuss the consistency of the estimators. I don't understand why I use correlation in a wrong way. No offensive but I want to know exactly where I go wrong Commented Aug 23, 2016 at 16:13