Why does the intercept column in model.matrix replace the first factor?

Question

I'm trying to convert my factor column to dummy variables:

str(cards$pointsBin)
# Factor w/ 5 levels ".lte100",".lte150",..: 3 2 3 1 4 4 2 2 4 4 ...

labels <- model.matrix(~ pointsBin, data=cards)

head(labels)

#     (Intercept) pointsBin.lte150 pointsBin.lte200 pointsBin.lte250 pointsBin.lte300
# 741           1                0                0                0                0
# 407           1                1                0                0                0
# 676           1                0                0                1                0
# 697           1                1                0                0                0
# 422           1                0                1                0                0
# 300           1                0                1                0                0

There is no column for the first value of my factor (".lte100"), which is what the first row should be categorized as. How do I get this data back? And what does the Intercept column that seems to be all 1's mean?

Answer 1

Consider the following:

require(mlbench)

data(HouseVotes84, package = "mlbench")
head(HouseVotes84)

labels <- model.matrix(~ V1, data=HouseVotes84)
head(labels)

labels1 <- model.matrix(~ V1+1, data=HouseVotes84)
head(labels1)

labels0 <- model.matrix(~ V1+0, data=HouseVotes84)
head(labels0)

labels_1 <- model.matrix(~ V1-1, data=HouseVotes84)
head(labels_1)

The first two commands are identical. The last two commands specifies not to produce the intercept and keeps the two dummy variables produced.

Answer 2

In statistics, when we have a factor variable with $k$ levels, we need to convert it to $k - 1$ indicator variables. We choose one level as the baseline, and then have an indicator variable for each of the remaining levels.

First let me explain why this isn't throwing away any information. Say there are levels A, B, C, and we have $I_B$ and $I_C$, the indicators for being in level B and C. An individual is in level A if and only if $I_B = 0$ (not in B) and $I_C = 0$ (not in C). So we still have kept track of the individuals in level A. This works for any number of levels.

Now as you note we could code the same information with three indicator variables $I_A$, $I_B$ and $I_C$. The reason why we don't is known as multicollinearity. In short, the columns of the regression matrix won't be linearly independent. This means the matrix $X^T X$ is not invertible, so we can't perform linear regression, as we need to calculate this inverse to calculate the regression estimates $\hat{\beta} = \left(X^T X\right)^{-1}X^T y$.

To take a really simple example, say we have two individuals, one in each of two levels A and B, and an intercept term in our regression.

The regression matrix $X$ will be \begin{equation} X = \begin{bmatrix} 1 & 1 & 0 \\ 1 & 0 & 1 \\ \end{bmatrix} \end{equation} Then \begin{align} X^T X & = \begin{bmatrix} 1 & 1\\ 1 & 0\\ 0 & 1\\ \end{bmatrix} \begin{bmatrix} 1 & 1 & 0 \\ 1 & 0 & 1 \\ \end{bmatrix}\\ & = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 1 & 0 \\ 1 & 0 & 1\\ \end{bmatrix} \end{align}

You can then see that adding the second and third columns of $X^T X$ gives the first column, so the inverse can't be computed.

This applies in general the Wikipedia page on multicollinearity gives more of an explanation.

If there is an exact linear relationship (perfect multicollinearity) among the independent variables, at least one of the columns of $X$ is a linear combination of the others, and so the rank of $X$ (and therefore of $X^T X$) is less than $k+1$, and the matrix $X^T X$ will not be invertible.

It is ok to include an indicator for all levels of the factor if you don't include the intercept term, and this encodes the exact same information. (The intercept term minus all other indicators gives the indicator for the baseline factor). But if you have more than one factor in the model this runs into the multicollinearity problem, so you can only include all levels if there is only one factor variable and no intercept term in the model.