# Co-variance of beta coefficients for Dummy Variable regression with intercept

If I have a dummy variable regression output with intercept included (base category as omitted category), and I have to do a hypothesis test for difference of means between two categories other than base category, I would need to set up the standard error as

$$\mathrm{SE}(\beta_i - \beta_j) = \sqrt{\mathrm{Var}(\beta_i) + \mathrm{Var}(\beta_j) - 2 \mathrm{Cov}(\beta_i, \beta_j)}$$

The $$\mathrm{Var}(\beta_i)$$ and $$\mathrm{Var}(\beta_j)$$ are reported in the regression output.

However, suppose I do not have the Covariance values reported in output because the var-covar matrix was not reported.

Is there a way that these co-variances between the betas (off-diagonal elements in the variance-covariance matrix) are directly calculated by knowing the diagonal elements and Mean Squared Error values?

## 1 Answer

Yes.

We have a dummy variable regression in which (without loss of generality) the last category serves as the base case, $$y_i=\alpha+\sum_{j=1}^{p-1}\beta_jD_{ij}+u_i$$ Let there be $n$ observations in total, with $n_j$ observations such that $D_{ij}=1$. We need to look at what happens to the formula for the variance of the regression coefficients, $\sigma^2(X'X)^{-1}$. Here, the regressor matrix $X$ has unit entries in the first column and another unit entry in column $j+1$ if observation $i$ belongs to group $j$ (unless it belongs to group $p$).

Consider $X'X$, which can be written as a block matrix $$X'X=\begin{pmatrix} A&B\\ B'&D \end{pmatrix}$$ where $A=n$, $B=(n_1,\cdots,n_{p-1})$ and $D$ a diagonal matrix with main diagonal $B$. This follows by direct multiplication, exploiting that no row of $X$ has more than one entry equal to one (except for the constant column)

To obtain $(X'X)^{-1}$, we use the formula for block inverses, \begin{align} (X'X)^{-1} &= \begin{pmatrix} \hspace{2cm}A &\hspace{4.3cm}B\\ \hspace{2cm}B' &\hspace{7cm}D \hspace{2.8cm}\end{pmatrix}^{-1} \\[5pt] &=\begin{pmatrix} (A-BD^{-1}B')^{-1}&-(A-BD^{-1}B')^{-1}BD^{-1}\\ -D^{-1}B'(A-BD^{-1}B')^{-1}&D^{-1}+D^{-1}B'(A-BD^{-1}B')^{-1}BD^{-1} \end{pmatrix} \end{align} The inverse of the diagonal matrix $D$ simply is a diagonal matrix with entries $1/n_j$. Direct multiplication then yields $$(A-BD^{-1}B')^{-1}=\left(n-\sum_{j=1}^{p-1}n_j\right)^{-1}=\frac{1}{n_p}$$ Further, $BD^{-1}=\iota'$, a unit row vector, and hence $D^{-1}B'=\iota$. Putting things together gives \begin{align} (X'X)^{-1} &=\begin{pmatrix} \hspace{.5cm}A &\hspace{.7cm}B\\ \hspace{.5cm}B' &\hspace{1.5cm}D \hspace{.8cm}\end{pmatrix}^{-1} \\[5pt] &=\begin{pmatrix} \frac{1}{n_p}&-\frac{1}{n_p}\iota'\\ -\frac{1}{n_p}\iota'&D^{-1}+\frac{1}{n_p}\iota\iota' \end{pmatrix} \end{align} This means that all off-diagonal elements of the variance-covariance matrix only depend on $1/n_p$, which you know from the first squared standard error.

In fact, the derivation hence shows a little more than what you asked: to get the off-diagonal elements, you do not even need to all variances, but only that of the base category.

Here is a little numerical illustration.

n <- 100
y <- rnorm(n)   # this is the dependent variable
p <- 5
X <- matrix(0, nrow=n, ncol=p)
X[cbind(1:n, sample(1:p, n, replace=T))] <- 1 # insert an 1 into one of the columns for each row
reg3 <- summary(lm(y~X[,1:(p-1)])) # regression omitting the pth category
vcov(reg3)

(Intercept) X[, 1:(p - 1)]1 X[, 1:(p - 1)]2 X[, 1:(p - 1)]3 X[, 1:(p - 1)]4
(Intercept)      0.05463828     -0.05463828     -0.05463828     -0.05463828     -0.05463828
X[, 1:(p - 1)]1 -0.05463828      0.14440116      0.05463828      0.05463828      0.05463828
X[, 1:(p - 1)]2 -0.05463828      0.05463828      0.13318080      0.05463828      0.05463828
X[, 1:(p - 1)]3 -0.05463828      0.05463828      0.05463828      0.10927656      0.05463828
X[, 1:(p - 1)]4 -0.05463828      0.05463828      0.05463828      0.05463828      0.10699996

• thanks a lot for the very detailed explanation and extremely useful code. It has helped a lot!!!
– vedo
Commented Sep 15, 2016 at 17:30