If I have a dummy variable regression output with intercept included (base category as omitted category), and I have to do a hypothesis test for difference of means between two categories other than base category, I would need to set up the standard error as

$$ \mathrm{SE}(\beta_i - \beta_j) = \sqrt{\mathrm{Var}(\beta_i) + \mathrm{Var}(\beta_j) - 2 \mathrm{Cov}(\beta_i, \beta_j)} $$

The $\mathrm{Var}(\beta_i)$ and $\mathrm{Var}(\beta_j)$ are reported in the regression output.

However, suppose I do not have the Covariance values reported in output because the var-covar matrix was not reported.

Is there a way that these co-variances between the betas (off-diagonal elements in the variance-covariance matrix) are directly calculated by knowing the diagonal elements and Mean Squared Error values?


1 Answer 1



We have a dummy variable regression in which (without loss of generality) the last category serves as the base case, $$ y_i=\alpha+\sum_{j=1}^{p-1}\beta_jD_{ij}+u_i $$ Let there be $n$ observations in total, with $n_j$ observations such that $D_{ij}=1$. We need to look at what happens to the formula for the variance of the regression coefficients, $\sigma^2(X'X)^{-1}$. Here, the regressor matrix $X$ has unit entries in the first column and another unit entry in column $j+1$ if observation $i$ belongs to group $j$ (unless it belongs to group $p$).

Consider $X'X$, which can be written as a block matrix $$ X'X=\begin{pmatrix} A&B\\ B'&D \end{pmatrix} $$ where $A=n$, $B=(n_1,\cdots,n_{p-1})$ and $D$ a diagonal matrix with main diagonal $B$. This follows by direct multiplication, exploiting that no row of $X$ has more than one entry equal to one (except for the constant column)

To obtain $(X'X)^{-1}$, we use the formula for block inverses, \begin{align} (X'X)^{-1} &= \begin{pmatrix} \hspace{2cm}A &\hspace{4.3cm}B\\ \hspace{2cm}B' &\hspace{7cm}D \hspace{2.8cm}\end{pmatrix}^{-1} \\[5pt] &=\begin{pmatrix} (A-BD^{-1}B')^{-1}&-(A-BD^{-1}B')^{-1}BD^{-1}\\ -D^{-1}B'(A-BD^{-1}B')^{-1}&D^{-1}+D^{-1}B'(A-BD^{-1}B')^{-1}BD^{-1} \end{pmatrix} \end{align} The inverse of the diagonal matrix $D$ simply is a diagonal matrix with entries $1/n_j$. Direct multiplication then yields $$ (A-BD^{-1}B')^{-1}=\left(n-\sum_{j=1}^{p-1}n_j\right)^{-1}=\frac{1}{n_p} $$ Further, $BD^{-1}=\iota'$, a unit row vector, and hence $D^{-1}B'=\iota$. Putting things together gives \begin{align} (X'X)^{-1} &=\begin{pmatrix} \hspace{.5cm}A &\hspace{.7cm}B\\ \hspace{.5cm}B' &\hspace{1.5cm}D \hspace{.8cm}\end{pmatrix}^{-1} \\[5pt] &=\begin{pmatrix} \frac{1}{n_p}&-\frac{1}{n_p}\iota'\\ -\frac{1}{n_p}\iota'&D^{-1}+\frac{1}{n_p}\iota\iota' \end{pmatrix} \end{align} This means that all off-diagonal elements of the variance-covariance matrix only depend on $1/n_p$, which you know from the first squared standard error.

In fact, the derivation hence shows a little more than what you asked: to get the off-diagonal elements, you do not even need to all variances, but only that of the base category.

Here is a little numerical illustration.

n <- 100
y <- rnorm(n)   # this is the dependent variable
p <- 5 
X <- matrix(0, nrow=n, ncol=p) 
X[cbind(1:n, sample(1:p, n, replace=T))] <- 1 # insert an 1 into one of the columns for each row
reg3 <- summary(lm(y~X[,1:(p-1)])) # regression omitting the pth category

                (Intercept) X[, 1:(p - 1)]1 X[, 1:(p - 1)]2 X[, 1:(p - 1)]3 X[, 1:(p - 1)]4
(Intercept)      0.05463828     -0.05463828     -0.05463828     -0.05463828     -0.05463828
X[, 1:(p - 1)]1 -0.05463828      0.14440116      0.05463828      0.05463828      0.05463828
X[, 1:(p - 1)]2 -0.05463828      0.05463828      0.13318080      0.05463828      0.05463828
X[, 1:(p - 1)]3 -0.05463828      0.05463828      0.05463828      0.10927656      0.05463828
X[, 1:(p - 1)]4 -0.05463828      0.05463828      0.05463828      0.05463828      0.10699996
  • 1
    $\begingroup$ thanks a lot for the very detailed explanation and extremely useful code. It has helped a lot!!! $\endgroup$
    – vedo
    Sep 15, 2016 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.