Confused about what a reference level is

Question

This is the definition of a reference level in my textbook: I don't understand why each level isn't given a coefficient? Why are we excluding a potential explanatory variable?

Answer 1

While it seems like we are excluding an explanatory variable, it turns out that it's still there but hidden in the intercept.

Let's do an example with two groups, each with three observations. (The reference in your question specifies $k>2$. What it says is, in fact, true for $k\ge2.$)

The typical way to write this model would be to have the follow regression equation:

$$ \hat{y} = \beta_0 + \beta_1x_1 $$

In this case, $x_1$ indicates if we are in the control group (0) or the treatment group (1).

We get the following model matrix:

$$ X = \begin{bmatrix} 1 & 0\\ 1 & 0\\ 1 & 0\\ 1 & 1\\ 1 & 1\\ 1 & 1 \end{bmatrix} $$

This matrix has full rank, so everything works when we do the OLS estimator $\hat{\beta} = (X^TX)^{-1}X^Ty$.

Your stance is that there are two groups, so we should have one variable indicating the control group and another indicating the treatment group:

$$ \hat{y} = \beta_0 + \beta_1x_{control} + \beta_2x_{treatment} $$

We get the following model matrix:

$$ X = \begin{bmatrix} 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & 0 & 1\\ 1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 1 & 0\\ \end{bmatrix} $$

Yikes! The second and third columns add up to the first! Then $X^TX$ is signular, so the usual OLS estimate, $\hat{\beta} = (X^TX)^{-1}X^Ty$, cannot be computed.

However, we got everything we needed without including the third column, so we are okay if we just do the model that way.

All of this assumes a column of $1$s in the model matrix, which corresponds to an intercept term in the regression equation. That intercept gives us a reference level. If we leave out the intercept, we get the following model:

$$ \hat{y} = \beta_1x_{control} + \beta_2x_{treatment} $$

This has the following model matrix:

$$ X = \begin{bmatrix} 0 & 1\\ 0 & 1\\ 0 & 1\\ 1 & 0\\ 1 & 0\\ 1 & 0\\ \end{bmatrix} $$

This matrix has full rank, so the usual OLS estimator works out.

Think about how you would do the parameter inference. Maybe our two groups have means wildly different from zero. Then the inference is not so useful. Sure, we confirm that the means of 6 gazillion and 6 gazillion plus 1 are different from zero, but we already knew that. What interests us is if the two groups have significantly different means. When we include an intercept term, the parameter inference tells us if the difference between treatment and control is significantly different from zero, precisely the question that is (usually) going to interest us.