80
$\begingroup$

There are two different ways to encoding categorical variables. Say, one categorical variable has n values. One-hot encoding converts it into n variables, while dummy encoding converts it into n-1 variables. If we have k categorical variables, each of which has n values. One hot encoding ends up with kn variables, while dummy encoding ends up with kn-k variables.

I hear that for one-hot encoding, intercept can lead to collinearity problem, which makes the model not sound. Someone call it "dummy variable trap".

My questions:

  1. Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website.

  2. Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?

$\endgroup$

3 Answers 3

50
$\begingroup$

Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website.

For an unregularized linear model with one-hot encoding, yes, you need to set the intercept to be false or else incur perfect collinearity. sklearn also allows for a ridge shrinkage penalty, and in that case it is not necessary, and in fact you should include both the intercept and all the levels. For dummy encoding you should include an intercept, unless you have standardized all your variables, in which case the intercept is zero.

Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?

The intercept is an additional degree of freedom, so in a well specified model it all equals out.

For the second one, what if there are k categorical variables? k variables are removed in dummy encoding. Is the degree of freedom still the same?

You could not fit a model in which you used all the levels of both categorical variables, intercept or not. For, as soon as you have one-hot-encoded all the levels in one variable in the model, say with binary variables $x_1, x_2, \ldots, x_n$, then you have a linear combination of predictors equal to the constant vector

$$ x_1 + x_2 + \cdots + x_n = 1 $$

If you then try to enter all the levels of another categorical $x'$ into the model, you end up with a distinct linear combination equal to a constant vector

$$ x_1' + x_2' + \cdots + x_k' = 1 $$

and so you have created a linear dependency

$$ x_1 + x_2 + \cdots x_n - x_1' - x_2' - \cdots - x_k' = 0$$

So you must leave out a level in the second variable, and everything lines up properly.

Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?

The second thing does not actually work. The $3 \times 4 = 12$ column design matrix you create will be singular. You need to remove three columns, one from each of three distinct categorical encodings, to recover non-singularity of your design.

$\endgroup$
5
  • $\begingroup$ Thanks. For the second one, what if there are k categorical variables? k variables are removed in dummy encoding. Is the degree of freedom still the same? $\endgroup$
    – Munichong
    Commented Jul 18, 2016 at 21:15
  • $\begingroup$ @ChongWang I edited a response to your comment into my answer. $\endgroup$ Commented Jul 19, 2016 at 2:15
  • $\begingroup$ Sorry, I get a little lost here. Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct? So here the DF of dummy encoding is 9-1 while the DF of one-hot encoding is 12. Am I correct? $\endgroup$
    – Munichong
    Commented Jul 19, 2016 at 18:35
  • $\begingroup$ @ChongWang Edited again. $\endgroup$ Commented Jul 19, 2016 at 18:53
  • 1
    $\begingroup$ @MatthewDrury I have same problem with linear_model in sklearn. After dummy encoding Decision Tree and KNN works ok but Linear Regression falls in singularity. I understand from your answer that I should remove a "level from second variable" but I don't know what it practically means? For example I have 3 numeric features and 3 categorical (manufacturer, model and fuel_type). Model is naturally dependable on manufacturer since one manufacturer can have n models. So how to proceed in this kind of common scenario if I want to use Linear Regression? $\endgroup$
    – Hrvoje
    Commented Dec 12, 2018 at 7:20
7
$\begingroup$

To add a little to @MatthewDrury's answer regarding this question:

Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?

We can examine what the design matrix would look like with and without an intercept by using model.matrix from R.

With an intercept:

> df <- expand.grid(w = letters[1:4], x = letters[5:8], y = letters[9:12])
> model.matrix(~ w + x + y, df)
   (Intercept) wb wc wd xf xg xh yj yk yl
1            1  0  0  0  0  0  0  0  0  0
2            1  1  0  0  0  0  0  0  0  0
3            1  0  1  0  0  0  0  0  0  0
4            1  0  0  1  0  0  0  0  0  0
5            1  0  0  0  1  0  0  0  0  0
6            1  1  0  0  1  0  0  0  0  0
7            1  0  1  0  1  0  0  0  0  0
8            1  0  0  1  1  0  0  0  0  0
9            1  0  0  0  0  1  0  0  0  0
10           1  1  0  0  0  1  0  0  0  0
11           1  0  1  0  0  1  0  0  0  0
12           1  0  0  1  0  1  0  0  0  0
13           1  0  0  0  0  0  1  0  0  0
14           1  1  0  0  0  0  1  0  0  0
15           1  0  1  0  0  0  1  0  0  0
16           1  0  0  1  0  0  1  0  0  0
17           1  0  0  0  0  0  0  1  0  0
18           1  1  0  0  0  0  0  1  0  0
19           1  0  1  0  0  0  0  1  0  0
20           1  0  0  1  0  0  0  1  0  0
21           1  0  0  0  1  0  0  1  0  0
22           1  1  0  0  1  0  0  1  0  0
23           1  0  1  0  1  0  0  1  0  0
24           1  0  0  1  1  0  0  1  0  0
25           1  0  0  0  0  1  0  1  0  0
26           1  1  0  0  0  1  0  1  0  0
27           1  0  1  0  0  1  0  1  0  0
28           1  0  0  1  0  1  0  1  0  0
29           1  0  0  0  0  0  1  1  0  0
30           1  1  0  0  0  0  1  1  0  0
31           1  0  1  0  0  0  1  1  0  0
32           1  0  0  1  0  0  1  1  0  0
33           1  0  0  0  0  0  0  0  1  0
34           1  1  0  0  0  0  0  0  1  0
35           1  0  1  0  0  0  0  0  1  0
36           1  0  0  1  0  0  0  0  1  0
37           1  0  0  0  1  0  0  0  1  0
38           1  1  0  0  1  0  0  0  1  0
39           1  0  1  0  1  0  0  0  1  0
40           1  0  0  1  1  0  0  0  1  0
41           1  0  0  0  0  1  0  0  1  0
42           1  1  0  0  0  1  0  0  1  0
43           1  0  1  0  0  1  0  0  1  0
44           1  0  0  1  0  1  0  0  1  0
45           1  0  0  0  0  0  1  0  1  0
46           1  1  0  0  0  0  1  0  1  0
47           1  0  1  0  0  0  1  0  1  0
48           1  0  0  1  0  0  1  0  1  0
49           1  0  0  0  0  0  0  0  0  1
50           1  1  0  0  0  0  0  0  0  1
51           1  0  1  0  0  0  0  0  0  1
52           1  0  0  1  0  0  0  0  0  1
53           1  0  0  0  1  0  0  0  0  1
54           1  1  0  0  1  0  0  0  0  1
55           1  0  1  0  1  0  0  0  0  1
56           1  0  0  1  1  0  0  0  0  1
57           1  0  0  0  0  1  0  0  0  1
58           1  1  0  0  0  1  0  0  0  1
59           1  0  1  0  0  1  0  0  0  1
60           1  0  0  1  0  1  0  0  0  1
61           1  0  0  0  0  0  1  0  0  1
62           1  1  0  0  0  0  1  0  0  1
63           1  0  1  0  0  0  1  0  0  1
64           1  0  0  1  0  0  1  0  0  1

Without an intercept:

> model.matrix(~ w + x + y - 1, df)
   wa wb wc wd xf xg xh yj yk yl
1   1  0  0  0  0  0  0  0  0  0
2   0  1  0  0  0  0  0  0  0  0
3   0  0  1  0  0  0  0  0  0  0
4   0  0  0  1  0  0  0  0  0  0
5   1  0  0  0  1  0  0  0  0  0
6   0  1  0  0  1  0  0  0  0  0
7   0  0  1  0  1  0  0  0  0  0
8   0  0  0  1  1  0  0  0  0  0
9   1  0  0  0  0  1  0  0  0  0
10  0  1  0  0  0  1  0  0  0  0
11  0  0  1  0  0  1  0  0  0  0
12  0  0  0  1  0  1  0  0  0  0
13  1  0  0  0  0  0  1  0  0  0
14  0  1  0  0  0  0  1  0  0  0
15  0  0  1  0  0  0  1  0  0  0
16  0  0  0  1  0  0  1  0  0  0
17  1  0  0  0  0  0  0  1  0  0
18  0  1  0  0  0  0  0  1  0  0
19  0  0  1  0  0  0  0  1  0  0
20  0  0  0  1  0  0  0  1  0  0
21  1  0  0  0  1  0  0  1  0  0
22  0  1  0  0  1  0  0  1  0  0
23  0  0  1  0  1  0  0  1  0  0
24  0  0  0  1  1  0  0  1  0  0
25  1  0  0  0  0  1  0  1  0  0
26  0  1  0  0  0  1  0  1  0  0
27  0  0  1  0  0  1  0  1  0  0
28  0  0  0  1  0  1  0  1  0  0
29  1  0  0  0  0  0  1  1  0  0
30  0  1  0  0  0  0  1  1  0  0
31  0  0  1  0  0  0  1  1  0  0
32  0  0  0  1  0  0  1  1  0  0
33  1  0  0  0  0  0  0  0  1  0
34  0  1  0  0  0  0  0  0  1  0
35  0  0  1  0  0  0  0  0  1  0
36  0  0  0  1  0  0  0  0  1  0
37  1  0  0  0  1  0  0  0  1  0
38  0  1  0  0  1  0  0  0  1  0
39  0  0  1  0  1  0  0  0  1  0
40  0  0  0  1  1  0  0  0  1  0
41  1  0  0  0  0  1  0  0  1  0
42  0  1  0  0  0  1  0  0  1  0
43  0  0  1  0  0  1  0  0  1  0
44  0  0  0  1  0  1  0  0  1  0
45  1  0  0  0  0  0  1  0  1  0
46  0  1  0  0  0  0  1  0  1  0
47  0  0  1  0  0  0  1  0  1  0
48  0  0  0  1  0  0  1  0  1  0
49  1  0  0  0  0  0  0  0  0  1
50  0  1  0  0  0  0  0  0  0  1
51  0  0  1  0  0  0  0  0  0  1
52  0  0  0  1  0  0  0  0  0  1
53  1  0  0  0  1  0  0  0  0  1
54  0  1  0  0  1  0  0  0  0  1
55  0  0  1  0  1  0  0  0  0  1
56  0  0  0  1  1  0  0  0  0  1
57  1  0  0  0  0  1  0  0  0  1
58  0  1  0  0  0  1  0  0  0  1
59  0  0  1  0  0  1  0  0  0  1
60  0  0  0  1  0  1  0  0  0  1
61  1  0  0  0  0  0  1  0  0  1
62  0  1  0  0  0  0  1  0  0  1
63  0  0  1  0  0  0  1  0  0  1
64  0  0  0  1  0  0  1  0  0  1

We can see that when we use an intercept, model.matrix uses dummy encoding with each variable w, x, and y being turned into 3 dummy variables, plus an intercept column. So there is a total of 10 degrees of freedom.

When we don't use an intercept, model.matrix creates 4 dummy variables for w and 3 dummy variables for x and y (and no intercept column). So the number of degrees of freedom is still 10.

$\endgroup$
1
  • $\begingroup$ +1 for the example. Made things clearer for me. $\endgroup$
    – irene
    Commented Jul 7, 2020 at 5:53
5
$\begingroup$

I totally agree with @Matthew Drury and @Cameron Bieganek's analysis of perfect collinearity and degree of freedom.

However, I want to argue here that we do not need to avoid perfect collinearity if we are using methods such as gradient descends as our optimizer. (Update, I just realized that there are more situations where we do not need to avoid collinearity: For example, We do not need to avoid perfect collinearity if we do regression with regularizer either. Since it will add a identity matrix $\lambda I$ to the matrix($X^TX$) below invertible again.)

The reason why we might want to avoid perfect collinearity is when we are using linear regression and our loss function is MSE, we could solve the closed form solution, which involves the inverse of a matrix about $X$, and perfect collinearity would make this matrix non invertible. However, in practice, as the computation of inverse matrix is quite expensive $\mathcal{O}(n^3)$, we could use other faster method such as gradient descend to compute an approximate solution and this process does not involves the inverse of matrix. Thus, perfect collinearity could be tolerated here.(Maybe this is why they do not have a warning) So, we could either use:

  1. one hot with intercept or
  2. one hot without intercept or
  3. dummy with intercept.

and they would generate quite similar result.

I run a regression using mpg dateset with sklearn, apply one hot or dummy to the "origin" feature which has three categories and the result is quite similar: (Please pay attention to how the residual and the coef are alike to each other, I marked parameters in front of enumerated category terms with red rectangles, with the others are the parameters of some continuous features. Detailed code could be seen here, sorry for the lack of comment and a large part of the code comes from tensorflow turorials.) one hot regression result dummy regression result

Relationship between the three results:

  1. BTW, we could also observe that for case of one hot without intercept, the parameters in front of the last three one hot features are actually equals to the element-wise sum of intercept and the parameters in front of the last three one hot features in one hot with intercept, which can be explained by the perfect collinearity.

  2. We can also notice that in the dummy with intercept case, the intercept is actually the parameter(-13.71778...) of the second categorical encoded term in the one-hot without intercept case. And the two parameter of the categorical encoded term in the dummy case is the difference between the corresponding parameters wrt the second term parameter, which is consistent with the interpretation of the parameter before categorical terms in econometrics: how much different each of the other categories makes to the output compare to the basic category.

$\endgroup$
2
  • $\begingroup$ I was looking for this addition. Also, not all ML models suffer from collinearity in the data. $\endgroup$
    – pietz
    Commented Apr 14, 2021 at 7:59
  • $\begingroup$ Thanks! This made my day! And I totally agree that not all ML models suffer from collinearity. (Tbh I can only think of close form solution for Linear Regression that actually suffers.) $\endgroup$
    – Flora Sun
    Commented Apr 15, 2021 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.