Qualitative variable coding in regression leads to "singularities"

Question

I have an independent variable called "quality"; this variable has 3 modalities of response (bad quality; medium quality; high quality). I want to introduce this independent variable into my multiple linear regression. When I have a binary independent variable (dummy variable, I can code 0 / 1) it is easy to introduce it into a multiple linear regression model.

But with 3 modalities of response, I have tried to code this variable like this :

Bad quality      Medium quality      High quality

     0                1                  0
     1                0                  0
     0                0                  1
     0                1                  0

But there is a problem when I try to do my multiple linear regression: the modality Medium quality gives me NA:

Coefficients: (1 not defined because of singularities) 

How can I code this variable "quality" with 3 modalities? Do I have to create a variable as a factor (factor in R) but then can I introduce this factor in a multiple linear regression?

Answer 1

The problem you are having (i.e., "singularities") can be thought of as an instance of multicollinearity. Multicollinearity is often defined as:

One or more predictor variables are a linear combination of other predictor variables.

This is, in fact, a rather strict definition; it is perfect multicollinearity, and you can easily have a problem with multicollinearity without any of your variables being perfect linear combinations of others. Moreover, perfect multicollinearity rarely occurs. However, you have stumbled across an case where it can occur. Let us see how we can perfectly predict medium quality from our knowledge of the other two categories (we'll do this with a regression model where medium quality is $Y$, and bad quality & high quality are $X_1$ & $X_2$, respectively):
$$ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 $$ Note that there is no error term, $\varepsilon$, specified, because we can predict this perfectly. To do so, we set $\beta_0 = 1$, $\beta_1 = -1$, and $\beta_2 = -1$. Now, when you have bad quality, then $X_1=1$, which cancels out $\beta_0$ ($1\; + \;-1\!\times\! 1$), and $X_2=0$ so that term is canceled out as well ($-1\times 0$). Thus, we are left with a predicted value of $0$ for $Y$ (medium quality), which is exactly correct. I will leave it to you to work out the other possibilities (it always works, in your case).

So what then should you do? When representing a categorical variable, we typically use reference cell coding (often called 'dummy coding'). To do this, we pick one level of our categorical variable as the reference level; that level does not get its own dummy code, but is simply indicated by having all $0$'s in the dummy codes for all other levels. The other levels of your categorical variable are represented by dummy codes just as you have already done. (For some more information on this, you can see my answer here: Regression based for example on days of week.) If you are using R, you can use a factor and R will do this all for you--it will be done correctly, and it's much more convenient--nonetheless, it's worth understanding that this is what is happening 'behind the scenes'.

Answer 2

@gung has explained the theory clearly. Here's a practical example to illustrate:

set.seed(1)
pred1 <- factor(c("bad", "med", "high"), levels=c("bad", "med", "high"))
df1 <- data.frame(y=20*abs(runif(6)),
                  x=rnorm(6),
                  q=sample(pred1, 6, replace=TRUE)
                  )
l1 <- lm(y ~ x, data=df1)
### add variable q    
l2 <- lm(y ~ x + q, data=df1)
### look at dummy variables generated in creating model
model.matrix(l2)

This shows us that the reference level (all $0$s) is bad as seen here in row 4:

  (Intercept)          x qmed qhigh
1           1  1.5952808    1     0
2           1  0.3295078    0     1
3           1 -0.8204684    0     1
4           1  0.4874291    0     0
5           1  0.7383247    1     0
6           1  0.5757814    0     0

Now if we code the dummy variables ourselves and try to fit a model using all of them:

df1 <- within(df1, {
       qbad <- ifelse(q=="bad", 1, 0)
       qmed <- ifelse(q=="med", 1, 0)
       qhigh <- ifelse(q=="high", 1, 0)
       })    
lm(y ~ x + qbad + qmed + qhigh, data=df1, singular.ok=FALSE)

We get the expected error: singular fit encountered