2
$\begingroup$

I have been wondering about this for a while. When I have mutually exclusive dummies in the same regression, are their coefficients independent?

For example, suppose I have the following:

$y_{i}=\mathbf{\mathbf{\alpha'}}\mathbf{x_{i}}+\beta_{0-5}I\left\{ z_{i}\in\left[0,5\right]\right\} +\beta_{6-10}I\left\{ z_{i}\in\left[6,10\right]\right\} +\beta_{11-15}I\left\{ z_{i}\in\left[11,15\right]\right\} +\epsilon_{i}$

where $\mathbf{x_{i}}$ is a vector of controls and $z_{i}\in\left\{ 0,1,2,3...15\right\}$ . In the end, I would like to test a non-linear combination of the coefficients $\beta_{0-5}$ , $\beta_{11-15}$ , and $\beta_{6-10}$ .

Here is my guess: $\hat{\beta}_{0-5}=f\left(Y,X|Z\in\left[0,5\right]\right)$ and $\hat{\beta}_{6-10}=f\left(Y,X|Z\in\left[6,10\right]\right)$ and so the random variables $\hat{\beta}_{0-5}$ and $\hat{\beta}_{6-10}$ are functions of different random variables, which leads me to think they are independent. But something doesn't seem right to me. Don't the $x_i$ and $y_i$ of all $z_{i}$ -categories affect the estimates of $\alpha$ ? And don't the estimates of $\alpha$ affect the estimates of the $\beta$ 's (e.g. the more $\alpha$ explains about the variance of $y_{i}$ , the less the $\beta$ 's will)?

Does this depend on the type of regression? I am interested in linear regression and ordered logit regression (where $y_{i}\in\left\{ 0,1,2,3,4,5\right\}$) .

I don't have a specific application in mind but I can come up with an example if that helps.

$\endgroup$
2
$\begingroup$

With large sample sizes it is reasonable to assume that linear regression coefficient estimates are normally distributed. If you can find a case where the covariance between the estimated coefficients is zero, then they are also independent. In linear regression, the covariance matrix of the estimated coefficients is given by $s^2\left(X^TX\right)^{-1}$, where $s^2$ is the MSE of the fitted model and $X$ is the design matrix associated with the regression.


An example of when the estimated regression coefficients will be independent is the following:

Let $x_1$, $x_2$, and $x_3$ be indicators representing membership in the groupings you've given in your question (if $0\leq z\leq5$, then $x_1=1$, otherwise $x_1=0$, etc.). Fit the model $Y=X\beta+\epsilon$, where $Y$ is an $n\times1$ column vector, $X$ is an $n\times3$ matrix whose columns are $x_1$, $x_2$, and $x_3$ (note: no intercept), and $epsilon$ is an $n\times1$ column vector, with the usual assumptions on the errors.

To calculate the covariance matrix of the estimated coefficients, let's first calculate $\left(X^TX\right)^{-1}$. It's easy to see that $X^TX$ gives the following $3\times3$ matrix: $$X^TX=\begin{bmatrix}n_1 & 0 & 0\\0 & n_2 & 0\\0 & 0 & n_3\end{bmatrix}$$ where $n_1$, $n_2$, and $n_3$ give the number of records in each group. $\left(X^TX\right)^{-1}$ is then $$\left(X^TX\right)^{-1}=\begin{bmatrix}1/n_1 & 0 & 0\\0 & 1/n_2 & 0\\0 & 0 & 1/n_3\end{bmatrix}$$ Multiplying by $s^2$ gives the covariance matrix of the estimated coefficients.

$$s^2\left(X^TX\right)^{-1}=\begin{bmatrix}s^2/n_1 & 0 & 0\\0 & s^2/n_2 & 0\\0 & 0 & s^2/n_3\end{bmatrix}$$

So in this case there is zero covariance between the estimated coefficients. For large sample sizes these zero covariances indicate independence of the estimated coefficients.


Now what happens when we add a single covariate?

The model is still $Y=X\beta+\epsilon$, but $X$ is now an $n\times4$ matrix, with the fourth column representing the new covariate. $X^TX$ is now

$$X^TX=\begin{bmatrix}n_1 & 0 & 0 & \langle x_1,x_4\rangle\\0 & n_2 & 0 & \langle x_2,x_4\rangle\\0 & 0 & n_3 & \langle x_3,x_4\rangle\\\langle x_1,x_4\rangle & \langle x_2,x_4\rangle & \langle x_3,x_4\rangle & \langle x_4,x_4\rangle\end{bmatrix}$$ where $\langle x_i,x_j\rangle$ is the dot product of columns $i$ and $j$ of $X$.

Inverting $X^TX$ is much harder in this case, but there is a special formula for inverting block matrices. I won't go through all of the matrix algebra here. In fact, I didn't even compute the full inverse; I computed just the upper left $3\times3$ block of the inverse.

$$\left(X^TX\right)_{1:3,1:3}^{-1}=\begin{bmatrix}1/n_1&0&0\\0&1/n_2&0\\0&0&1/n_3\end{bmatrix}+\frac{1}{\langle x_4,x_4\rangle-\sum_{i=1}^{3}n_i^{-1}\langle x_i,x_4\rangle^2}G$$ where $G$ is $$G=\begin{bmatrix}n_1^{-2}\langle x_1,x_4\rangle^2&(n_1n_2)^{-1}\langle x_1,x_4\rangle\langle x_2,x_4\rangle&(n_1n_3)^{-1}\langle x_1,x_4\rangle\langle x_3,x_4\rangle\\(n_1n_2)^{-1}\langle x_1,x_4\rangle\langle x_2,x_4\rangle&n_2^{-2}\langle x_2,x_4\rangle^2&(n_2n_3)^{-1}\langle x_2,x_4\rangle\langle x_3,x_4\rangle\\(n_1n_3)^{-1}\langle x_1,x_4\rangle\langle x_3,x_4\rangle&(n_2n_3)^{-1}\langle x_2,x_4\rangle\langle x_3,x_4\rangle&n_3^{-2}\langle x_3,x_4\rangle^2\end{bmatrix}$$

Notice that the estimated coefficients on $x_i$ and $x_j$ will have zero covariance (and with large sample sizes, independence) when at least one of $\langle x_i,x_4\rangle$ or $\langle x_j,x_4 \rangle$ is zero.


That's about all I can handle to write at the moment.

$\endgroup$
  • $\begingroup$ Thanks for your answer and explanation. This gives me some good things to study and try to squeeze out intuition from. My takeaway for the moment is: if no covariates, then yes they are independent (you showed the math, and more intuitively, they are just mutually independent averages of the dependent). When there are covariates,in general, there is no independence. $\endgroup$ – Roberta Jankowski Nov 5 '12 at 1:52
  • $\begingroup$ If you do the general case later, I will study it. Only do it if it would be fun for you, but I wanted to let you know that it would not be wasted effort. Thanks! $\endgroup$ – Roberta Jankowski Nov 5 '12 at 1:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.