22
$\begingroup$

I am not sure why we need to dummy code categorical variables. For instance, if I have a categorical variable with four possible values 0,1,2,3 I can replace it by two dimensions. If the variable had value 0, it would have 0,0 in the two dimension, if it had 3, it would have 1,1 in the two dimension and so on.

I am not sure why we need to do this?

$\endgroup$
21
$\begingroup$

Suppose your four categories are eye colors (code): brown (1), blue (2), green (3), hazel (4)—ignoring heterochromia, violet, red, gray, etc. for the moment.

In no way (that I can currently imagine) would we mean that green $ = 3\times $ brown, or that hazel $=2\times$ blue as our codes imply, even though $3=3\times1$ and $4 = 2 \times 2$.

Therefore (unless we for some reason do want such meaning to slip into our analyses), we need to use some sort of coding. Dummy coding is one example, which eliminates such relationships from the statistical stories we want to tell about eye color. Effect coding and Heckman coding are other examples.

Update: your example of two variables for four categories does not match my understanding use of the term "dummy code" which typically entails replacing $k$ categories (say 4) with $k-1$ dummy variables (sorting observations by category):

id  category  dummy1 dummy2 dummy3
 1         1       1      0      0
 2         1       1      0      0
 3         2       0      1      0
 4         2       0      1      0
 5         3       0      0      1
 6         3       0      0      1
 7         4       0      0      0
 8         4       0      0      0

Here category 4 is the reference category, assuming that there is a constant in your model, such as:

$$y = \beta_{0} + \beta_{1}d1 + \beta_{2}d2 + \beta_{3}d3 + \varepsilon$$

where $\beta_{0}$ is the mean value of $y$ when category = 4, and the $\beta$ terms associated with each dummy indicate by what amount $y$ changes from $\beta_{0}$ for that category.

If you do not have a constant ($\beta_{0}$) term in the model, then you need one more "dummy" predictor (perhaps less often termed "indicator variables"), in effect the dummies then each behave as the model constant for each category:

$$y = \beta_{1}d1 + \beta_{2}d2 + \beta_{3}d3 + \beta_{4}d4 + \varepsilon$$

So this would get one around the issue of creating nonsensical quantitative relationships between category codes I mention at first, but why not use user12331-coding as you suggest? user12331-coding candidate A:

id  category   code1  code2
 1         1       0      ?
 2         1       0      ?
 3         2       1      ?
 4         2       1      ?
 5         3       ?      0
 6         3       ?      0
 7         4       ?      1
 8         4       ?      1

you are quite right to point out that one can represent 4 values using 2 binary variables (i.e. two-bits). Unfortunately, one approach to this (code1 for categories 1 and 2, and code2 for categories 3 and 4) leaves the ambiguity indicated by the question marks: what values would go there?!

Well, what about a second approach, call it user12331-coding candidate B:

id  category   code1  code2
 1         1       0      0
 2         1       0      0
 3         2       0      1
 4         2       0      1
 5         3       1      0
 6         3       1      0
 7         4       1      1
 8         4       1      1

There! No ambiguity, right? Right! Unfortunately, all this coding does is represent the numerical quantities 1–4 (or 0–3) in binary notation, which leaves intact the problem of giving those undesired quantitative relationships to the categories.

Hence, the need for another coding scheme.

I will close with the caveat that the various coding schemes are more or less a matter or style (i.e. what does one want a specific $\beta$ to mean) unless one also includes interaction terms with the categories in the model. Then dummy coding will induce an artificial heteroscedasticity and bias the standard errors, so you would want to stick with effect coding in such a case (there may be other coding systems that keep one safe in that circumstance, but I am unfamiliar with them).

$\endgroup$
  • 5
    $\begingroup$ While this answer demonstrates the reason why we can't use one variable (i.e. that we do indeed need 'some sort of coding'), it doesn't (yet) explain why we can't do it with say two variables, as the OP suggests in the question. $\endgroup$ – Glen_b -Reinstate Monica Sep 11 '14 at 2:15
  • $\begingroup$ @Glen_b Thank you. I hope my update has helped address. $\endgroup$ – Alexis Sep 11 '14 at 16:02
  • 2
    $\begingroup$ Note that 2 binary variables is sufficient to represent 4 categories [(0,0), (0,1), (1,0), (1,1)], but is not the appropriate way to dummy code for analysis. The OP seems to be coding incorrectly. $\endgroup$ – Ellis Valentiner Sep 11 '14 at 16:23
  • $\begingroup$ @user12202013 Yes. As in my last example. $\endgroup$ – Alexis Sep 11 '14 at 19:18
  • $\begingroup$ What if I were to do a binary coding using two variables as suggested by OP, but then if the goal is prediction, then wouldn't a non-parametric, non-linear classifier/regressor work equally well? $\endgroup$ – tool.ish Apr 13 '17 at 12:32
1
$\begingroup$

My take on this question is, that coding the four possible states with just two variables is less expressive with some machine learning algorithms than using 4 variables.

For example, imagine that you want to do linear regression and your true mapping maps the values 0,1 and 2 to 0 and the value 3 to 1. You can quickly check that there is no way of learning this mapping with linear regression when coding your categorial variable with just two binary ones (just try to fit the corresponding plane in your head). On the other hand, when you use a 1-Of-K coding, this would not be a problem.

$\endgroup$
0
$\begingroup$

Your alternative is also a dummy code. You choose the dummy code that best expresses the relationship to your dependent variable. Eg colour could be expressed as 1 of n, or you could turn into numeric rgb components, or you could categorise: girly/muddy/...1 of n basically means each instance is learnt separately which is good if there is no relationship. .. but where there is a relationship you are wasting your data..you have to separately estimate the coefficient for each instance of the category...consider job as a categorical variable. You might re categorise as market sector and seniority.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.