Standard errors of regression coefficients in a Dummy Variable regression model

A dummy variable regression is equivalent to an ANOVA, and the beta coeffns are equal to the means of particular category with respect to the base category.

However, I am unable to interpret the "standard errors" of the beta coeffns se(beta1) etc. What do the standard errors mean and how are they calculated? Are these standard errors also related to the standard deviation of the sample for the particular category?

• There is nothing special about this standard errors! They are interpreted the same way as other standard erors. – kjetil b halvorsen Sep 13 '16 at 10:57
• Another fun fact: If you do robust standard errors, only have dummies variables, and have a saturated model (i.e. full interactions), then your standard errors are basically equivalent to what you would get by calculating the standard error of each category mean individually. – Matthew Gunn Sep 13 '16 at 14:20

While @Kjetil is of course right that there is nothing special about s.e.s in a dummy variable regression, it may be instructive to look at how the expressions look like explicitly.

Take the model (which slightly differs from yours in that there is a full set of dummies rather than an intercept and dummies for all but one category) $$y_i=\beta_1D_{1i}+\beta_2D_{2i}+u_i$$ where the dummies are such that $D_{1i}+D_{2i}=1$ for all $i=1,\ldots,n$.

Let there be $n_1$ observations such that $D_{1i}=1$ and $n_2$ such that $D_{2i}=1$. Then, the formula for the variance of the regression coefficients, $\sigma^2(X'X)^{-1}$, simplifies to $$Var(\hat\beta)=\begin{pmatrix}\sigma^2/n_1&0\\0&\sigma^2/n_2\end{pmatrix},$$ which are indeed nothing but the respective variances of the sample means of the $y_i$ belonging to the two different groups. The off-diagonal entries must be zero as the second regressor always has a zero entry when the first has a unit entry, so when computing the off-diagonal entries of $X'X$, we must multiply zeros and ones.

When we have one unit column and one dummy (here, $D_{1i}$), $X'X$ will become $$\begin{pmatrix}n&n_1\\n_1&n_1\end{pmatrix}$$ so that $$(X'X)^{-1}=\frac{1}{nn_1-n_1^2}\begin{pmatrix}n_1&-n_1\\-n_1&n\end{pmatrix}$$ or $$(X'X)^{-1}=\begin{pmatrix}\frac{1}{n_2}&-\frac{1}{n_2}\\-\frac{1}{n_2}&\frac{n}{nn_1-n_1^2}\end{pmatrix}$$ Hence, the off-diagonal is no longer zero, but basically cancels out the variance of the baseline category. I am not so sure what to make of this finding, but note that this implies that the variance of the sum of coefficients in the regression with intercept, $$Var(\hat\beta_0+\hat\beta_1)=\frac{1}{n-n_1}-2\frac{1}{n-n_1}+\frac{n}{nn_1-n_1^2},$$ equals the variance of $D_{1i}$, $1/n_1$, in the regression with two dummies.

Here is a little numerical illustration of these ideas.

n <- 1000

y <- rnorm(n)                   # some dependent variable
x1 <- rbinom(n, size=1, p=.4)   # a dummy regressor
x2 <- 1-x1                      # its complement

(reg1 <- summary(lm(y~x1+x2-1)))# the regression with full dummies
n1 <- length(y[x1==1])          # the y's belonging to the first regressor
n2 <- length(y[x2==1])          # the y's belonging to the second regressor
mean(y[x1==1])                  # the means are indeed the point estimates
mean(y[x1==0])

reg1$sigma*sqrt(1/n1) # reproduces the standard errors reg1$sigma*sqrt(1/n2)
sd(y[x1==1])*sqrt(1/n1)         # not exactly the same because the regression uses a "pooled" estimator for sigma^2_u
vcov(reg1)                      # off-diagonal element is zero

(reg2 <- summary(lm(y~x1)))     # point estimate of the baseline category unaffected, also its standard error, but that of remaining dummy has changed
vcov(reg2)                      # now, the off-diagonal entry is no longer zero
sqrt(diag(vcov(reg2)))          # another way to look at standard errors

• thanks @ChristophHanck for the very detailed explanation. My doubt was exactly that. Because I want to relate my standard errors of the Beta coefficient with the standard deviation seen in the sample for that particular category. From the Variance-Covariance matrix that you have shown, can we also say that the co-variance between the dummies D1 and D2 will always be zero since they are different categories of the same variable? and can I extend this assumption also to a model that would have an intercept? I would be very grateful if you could shed some light on that as well. – vedo Sep 13 '16 at 12:42
• @vedo, I hope the edit addresses the remaining questions. – Christoph Hanck Sep 13 '16 at 13:32
• that helps a lot thank you very much. I just had one last question regarding this topic: If I have a dummy variable regression output with intercept included, and I have to do a hypothesis test for difference of means between two categories which are not base category, I would need to set up the standard error for "beta i - betaj " = sqrt(var(beta i) + var(beta j) - 2cov(beta i, beta j)). Suppose I do not have the variance-covariance matrix information, can the cov(beta i, beta j) be calculated by knowing only the standard errors for all the betas in the model? – vedo Sep 14 '16 at 13:59
• I suggest you ask this as a separate question. – Christoph Hanck Sep 14 '16 at 14:01
• Thanks a ton I will do so. However I just realized that all the covariance terms in this matrix (for dummy variable regression with intercept) are same with only negative sign in the first row and column. And this value of co-variance is nothing but the variance of sampling distribution of means of base category using pooled sigma-squared equal to value of Mean Squared Error. So if I know the standard error of base category(intercept), I can just square the same value to populate the co-variance terms while keeping in mind the +/- sign. Is that right? Just a Yes or No will do – vedo Sep 14 '16 at 18:47