17
$\begingroup$

I have an independent variable called "quality"; this variable has 3 modalities of response (bad quality; medium quality; high quality). I want to introduce this independent variable into my multiple linear regression. When I have a binary independent variable (dummy variable, I can code 0 / 1) it is easy to introduce it into a multiple linear regression model.

But with 3 modalities of response, I have tried to code this variable like this :

Bad quality      Medium quality      High quality

     0                1                  0
     1                0                  0
     0                0                  1
     0                1                  0

But there is a problem when I try to do my multiple linear regression: the modality Medium quality gives me NA:

Coefficients: (1 not defined because of singularities) 

How can I code this variable "quality" with 3 modalities? Do I have to create a variable as a factor (factor in R) but then can I introduce this factor in a multiple linear regression?

$\endgroup$
  • 2
    $\begingroup$ Perhaps I am misunderstanding the question, but you can't include all levels of a categorical variable into a linear regression because of perfect collinearity. One of your categories will be dropped to provide a base group against which the other groups are compared. $\endgroup$ – RickyB Sep 22 '13 at 15:21
  • 1
    $\begingroup$ There is an outstanding explanation in a statistical context of what constitutes a singular matrix here: what-correlation-makes-a-matrix-singular? $\endgroup$ – gung - Reinstate Monica Sep 24 '13 at 13:40
23
$\begingroup$

The problem you are having (i.e., "singularities") can be thought of as an instance of multicollinearity. Multicollinearity is often defined as:

One or more predictor variables are a linear combination of other predictor variables.

This is, in fact, a rather strict definition; it is perfect multicollinearity, and you can easily have a problem with multicollinearity without any of your variables being perfect linear combinations of others. Moreover, perfect multicollinearity rarely occurs. However, you have stumbled across an case where it can occur. Let us see how we can perfectly predict medium quality from our knowledge of the other two categories (we'll do this with a regression model where medium quality is $Y$, and bad quality & high quality are $X_1$ & $X_2$, respectively):
$$ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 $$ Note that there is no error term, $\varepsilon$, specified, because we can predict this perfectly. To do so, we set $\beta_0 = 1$, $\beta_1 = -1$, and $\beta_2 = -1$. Now, when you have bad quality, then $X_1=1$, which cancels out $\beta_0$ ($1\; + \;-1\!\times\! 1$), and $X_2=0$ so that term is canceled out as well ($-1\times 0$). Thus, we are left with a predicted value of $0$ for $Y$ (medium quality), which is exactly correct. I will leave it to you to work out the other possibilities (it always works, in your case).

So what then should you do? When representing a categorical variable, we typically use reference cell coding (often called 'dummy coding'). To do this, we pick one level of our categorical variable as the reference level; that level does not get its own dummy code, but is simply indicated by having all $0$'s in the dummy codes for all other levels. The other levels of your categorical variable are represented by dummy codes just as you have already done. (For some more information on this, you can see my answer here: Regression based for example on days of week.) If you are using R, you can use a factor and R will do this all for you--it will be done correctly, and it's much more convenient--nonetheless, it's worth understanding that this is what is happening 'behind the scenes'.

$\endgroup$
  • $\begingroup$ Thanks for all your comments ! I am using R so if I understood well, with a factor R will do everything, I don't need to do anything ! Perfect ! Tanks once more ! $\endgroup$ – varin sacha Sep 22 '13 at 20:42
  • $\begingroup$ If you set the intercept to zero in the lm formula (+ 0) would it work? $\endgroup$ – Firebug Jun 14 '16 at 19:29
  • 1
    $\begingroup$ @Firebug, if you are using level means coding (ie, you have a separate variable for every level), then you can set the intercept to zero & it should work fine. Although the meaning of the variable coefficients & the hypothesis tests will differ. $\endgroup$ – gung - Reinstate Monica Jun 14 '16 at 20:04
10
$\begingroup$

@gung has explained the theory clearly. Here's a practical example to illustrate:

set.seed(1)
pred1 <- factor(c("bad", "med", "high"), levels=c("bad", "med", "high"))
df1 <- data.frame(y=20*abs(runif(6)),
                  x=rnorm(6),
                  q=sample(pred1, 6, replace=TRUE)
                  )
l1 <- lm(y ~ x, data=df1)
### add variable q    
l2 <- lm(y ~ x + q, data=df1)
### look at dummy variables generated in creating model
model.matrix(l2)

This shows us that the reference level (all $0$s) is bad as seen here in row 4:

  (Intercept)          x qmed qhigh
1           1  1.5952808    1     0
2           1  0.3295078    0     1
3           1 -0.8204684    0     1
4           1  0.4874291    0     0
5           1  0.7383247    1     0
6           1  0.5757814    0     0

Now if we code the dummy variables ourselves and try to fit a model using all of them:

df1 <- within(df1, {
       qbad <- ifelse(q=="bad", 1, 0)
       qmed <- ifelse(q=="med", 1, 0)
       qhigh <- ifelse(q=="high", 1, 0)
       })    
lm(y ~ x + qbad + qmed + qhigh, data=df1, singular.ok=FALSE)

We get the expected error: singular fit encountered

$\endgroup$
  • 1
    $\begingroup$ Pleasure. Hope it's all clear now. factor will generally take care of dummy variable coding for you but good to be aware what's going on 'under the hood'. $\endgroup$ – dardisco Sep 23 '13 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.