# Independence of mutually exclusive dummies in same regression?

I have been wondering about this for a while. When I have mutually exclusive dummies in the same regression, are their coefficients independent?

For example, suppose I have the following:

$y_{i}=\mathbf{\mathbf{\alpha'}}\mathbf{x_{i}}+\beta_{0-5}I\left\{ z_{i}\in\left[0,5\right]\right\} +\beta_{6-10}I\left\{ z_{i}\in\left[6,10\right]\right\} +\beta_{11-15}I\left\{ z_{i}\in\left[11,15\right]\right\} +\epsilon_{i}$

where $\mathbf{x_{i}}$ is a vector of controls and $z_{i}\in\left\{ 0,1,2,3...15\right\}$ . In the end, I would like to test a non-linear combination of the coefficients $\beta_{0-5}$ , $\beta_{11-15}$ , and $\beta_{6-10}$ .

Here is my guess: $\hat{\beta}_{0-5}=f\left(Y,X|Z\in\left[0,5\right]\right)$ and $\hat{\beta}_{6-10}=f\left(Y,X|Z\in\left[6,10\right]\right)$ and so the random variables $\hat{\beta}_{0-5}$ and $\hat{\beta}_{6-10}$ are functions of different random variables, which leads me to think they are independent. But something doesn't seem right to me. Don't the $x_i$ and $y_i$ of all $z_{i}$ -categories affect the estimates of $\alpha$ ? And don't the estimates of $\alpha$ affect the estimates of the $\beta$ 's (e.g. the more $\alpha$ explains about the variance of $y_{i}$ , the less the $\beta$ 's will)?

Does this depend on the type of regression? I am interested in linear regression and ordered logit regression (where $y_{i}\in\left\{ 0,1,2,3,4,5\right\}$) .

I don't have a specific application in mind but I can come up with an example if that helps.

With large sample sizes it is reasonable to assume that linear regression coefficient estimates are normally distributed. If you can find a case where the covariance between the estimated coefficients is zero, then they are also independent. In linear regression, the covariance matrix of the estimated coefficients is given by $s^2\left(X^TX\right)^{-1}$, where $s^2$ is the MSE of the fitted model and $X$ is the design matrix associated with the regression.

An example of when the estimated regression coefficients will be independent is the following:

Let $x_1$, $x_2$, and $x_3$ be indicators representing membership in the groupings you've given in your question (if $0\leq z\leq5$, then $x_1=1$, otherwise $x_1=0$, etc.). Fit the model $Y=X\beta+\epsilon$, where $Y$ is an $n\times1$ column vector, $X$ is an $n\times3$ matrix whose columns are $x_1$, $x_2$, and $x_3$ (note: no intercept), and $epsilon$ is an $n\times1$ column vector, with the usual assumptions on the errors.

To calculate the covariance matrix of the estimated coefficients, let's first calculate $\left(X^TX\right)^{-1}$. It's easy to see that $X^TX$ gives the following $3\times3$ matrix: $$X^TX=\begin{bmatrix}n_1 & 0 & 0\\0 & n_2 & 0\\0 & 0 & n_3\end{bmatrix}$$ where $n_1$, $n_2$, and $n_3$ give the number of records in each group. $\left(X^TX\right)^{-1}$ is then $$\left(X^TX\right)^{-1}=\begin{bmatrix}1/n_1 & 0 & 0\\0 & 1/n_2 & 0\\0 & 0 & 1/n_3\end{bmatrix}$$ Multiplying by $s^2$ gives the covariance matrix of the estimated coefficients.

$$s^2\left(X^TX\right)^{-1}=\begin{bmatrix}s^2/n_1 & 0 & 0\\0 & s^2/n_2 & 0\\0 & 0 & s^2/n_3\end{bmatrix}$$

So in this case there is zero covariance between the estimated coefficients. For large sample sizes these zero covariances indicate independence of the estimated coefficients.

Now what happens when we add a single covariate?

The model is still $Y=X\beta+\epsilon$, but $X$ is now an $n\times4$ matrix, with the fourth column representing the new covariate. $X^TX$ is now

$$X^TX=\begin{bmatrix}n_1 & 0 & 0 & \langle x_1,x_4\rangle\\0 & n_2 & 0 & \langle x_2,x_4\rangle\\0 & 0 & n_3 & \langle x_3,x_4\rangle\\\langle x_1,x_4\rangle & \langle x_2,x_4\rangle & \langle x_3,x_4\rangle & \langle x_4,x_4\rangle\end{bmatrix}$$ where $\langle x_i,x_j\rangle$ is the dot product of columns $i$ and $j$ of $X$.

Inverting $X^TX$ is much harder in this case, but there is a special formula for inverting block matrices. I won't go through all of the matrix algebra here. In fact, I didn't even compute the full inverse; I computed just the upper left $3\times3$ block of the inverse.

$$\left(X^TX\right)_{1:3,1:3}^{-1}=\begin{bmatrix}1/n_1&0&0\\0&1/n_2&0\\0&0&1/n_3\end{bmatrix}+\frac{1}{\langle x_4,x_4\rangle-\sum_{i=1}^{3}n_i^{-1}\langle x_i,x_4\rangle^2}G$$ where $G$ is $$G=\begin{bmatrix}n_1^{-2}\langle x_1,x_4\rangle^2&(n_1n_2)^{-1}\langle x_1,x_4\rangle\langle x_2,x_4\rangle&(n_1n_3)^{-1}\langle x_1,x_4\rangle\langle x_3,x_4\rangle\\(n_1n_2)^{-1}\langle x_1,x_4\rangle\langle x_2,x_4\rangle&n_2^{-2}\langle x_2,x_4\rangle^2&(n_2n_3)^{-1}\langle x_2,x_4\rangle\langle x_3,x_4\rangle\\(n_1n_3)^{-1}\langle x_1,x_4\rangle\langle x_3,x_4\rangle&(n_2n_3)^{-1}\langle x_2,x_4\rangle\langle x_3,x_4\rangle&n_3^{-2}\langle x_3,x_4\rangle^2\end{bmatrix}$$

Notice that the estimated coefficients on $x_i$ and $x_j$ will have zero covariance (and with large sample sizes, independence) when at least one of $\langle x_i,x_4\rangle$ or $\langle x_j,x_4 \rangle$ is zero.

That's about all I can handle to write at the moment.

• Thanks for your answer and explanation. This gives me some good things to study and try to squeeze out intuition from. My takeaway for the moment is: if no covariates, then yes they are independent (you showed the math, and more intuitively, they are just mutually independent averages of the dependent). When there are covariates,in general, there is no independence. – Roberta Jankowski Nov 5 '12 at 1:52
• If you do the general case later, I will study it. Only do it if it would be fun for you, but I wanted to let you know that it would not be wasted effort. Thanks! – Roberta Jankowski Nov 5 '12 at 1:53