In statistics, when we have a factor variable with $k$ levels, we need to convert it to $k - 1$ indicator variables. We choose one level as the baseline, and then have an indicator variable for each of the remaining levels.
First let me explain why this isn't throwing away any information.
Say there are levels A, B, C, and we have $I_B$ and $I_C$, the indicators for being in level B and C. An individual is in level A if and only if $I_B = 0$ (not in B) and $I_C = 0$ (not in C). So we still have kept track of the individuals in level A. This works for any number of levels.
Now as you note we could code the same information with three indicator variables $I_A$, $I_B$ and $I_C$. The reason why we don't is known as multicollinearity. In short, the columns of the regression matrix won't be linearly independent. This means the matrix $X^T X$ is not invertible, so we can't perform linear regression, as we need to calculate this inverse to calculate the regression estimates $\hat{\beta} = \left(X^T X\right)^{-1}X^T y$.
To take a really simple example, say we have two individuals, one in each of two levels A and B, and an intercept term in our regression.
The regression matrix $X$ will be
\begin{equation}
X =
\begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 1 \\
\end{bmatrix}
\end{equation}
Then
\begin{align}
X^T X & =
\begin{bmatrix}
1 & 1\\
1 & 0\\
0 & 1\\
\end{bmatrix}
\begin{bmatrix}
1 & 1 & 0 \\
1 & 0 & 1 \\
\end{bmatrix}\\
& =
\begin{bmatrix}
2 & 1 & 1 \\
1 & 1 & 0 \\
1 & 0 & 1\\
\end{bmatrix}
\end{align}
You can then see that adding the second and third columns of $X^T X$ gives the first column, so the inverse can't be computed.
This applies in general the Wikipedia page on multicollinearity gives more of an explanation.
If there is an exact linear relationship (perfect multicollinearity) among the independent variables, at least one of the columns of $X$ is a linear combination of the others, and so the rank of $X$ (and therefore of $X^T X$) is less than $k+1$, and the matrix $X^T X$ will not be invertible.
It is ok to include an indicator for all levels of the factor if you don't include the intercept term, and this encodes the exact same information. (The intercept term minus all other indicators gives the indicator for the baseline factor). But if you have more than one factor in the model this runs into the multicollinearity problem, so you can only include all levels if there is only one factor variable and no intercept term in the model.
- 1
when you have a single categorical predictor represented as a factor (as in this question), it'll assume you don't really mean that & switch to using sum-to-zero coding; which is of course just a different parametrization. Too friendly, if you ask me. $\endgroup$