One-hot vs dummy encoding in Scikit-learn

Question

There are two different ways to encoding categorical variables. Say, one categorical variable has n values. One-hot encoding converts it into n variables, while dummy encoding converts it into n-1 variables. If we have k categorical variables, each of which has n values. One hot encoding ends up with kn variables, while dummy encoding ends up with kn-k variables.

I hear that for one-hot encoding, intercept can lead to collinearity problem, which makes the model not sound. Someone call it "dummy variable trap".

My questions:

1. Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website.

2. Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?

Answer 1

Scikit-learn's linear regression model allows users to disable intercept. So for one-hot encoding, should I always set fit_intercept=False? For dummy encoding, fit_intercept should always be set to True? I do not see any "warning" on the website.

For an unregularized linear model with one-hot encoding, yes, you need to set the intercept to be false or else incur perfect collinearity. sklearn also allows for a ridge shrinkage penalty, and in that case it is not necessary, and in fact you should include both the intercept and all the levels. For dummy encoding you should include an intercept, unless you have standardized all your variables, in which case the intercept is zero.

Since one-hot encoding generates more variables, does it have more degree of freedom than dummy encoding?

The intercept is an additional degree of freedom, so in a well specified model it all equals out.

For the second one, what if there are k categorical variables? k variables are removed in dummy encoding. Is the degree of freedom still the same?

You could not fit a model in which you used all the levels of both categorical variables, intercept or not. For, as soon as you have one-hot-encoded all the levels in one variable in the model, say with binary variables $x_1, x_2, \ldots, x_n$, then you have a linear combination of predictors equal to the constant vector

$$ x_1 + x_2 + \cdots + x_n = 1 $$

If you then try to enter all the levels of another categorical $x'$ into the model, you end up with a distinct linear combination equal to a constant vector

$$ x_1' + x_2' + \cdots + x_k' = 1 $$

and so you have created a linear dependency

$$ x_1 + x_2 + \cdots x_n - x_1' - x_2' - \cdots - x_k' = 0$$

So you must leave out a level in the second variable, and everything lines up properly.

Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?

The second thing does not actually work. The $3 \times 4 = 12$ column design matrix you create will be singular. You need to remove three columns, one from each of three distinct categorical encodings, to recover non-singularity of your design.

Answer 2

To add a little to @MatthewDrury's answer regarding this question:

Say, I have 3 categorical variables, each of which has 4 levels. In dummy encoding, 3*4-3=9 variables are built with one intercept. In one-hot encoding, 3*4=12 variables are built without an intercept. Am I correct?

We can examine what the design matrix would look like with and without an intercept by using model.matrix from R.

With an intercept:

> df <- expand.grid(w = letters[1:4], x = letters[5:8], y = letters[9:12])
> model.matrix(~ w + x + y, df)
   (Intercept) wb wc wd xf xg xh yj yk yl
1            1  0  0  0  0  0  0  0  0  0
2            1  1  0  0  0  0  0  0  0  0
3            1  0  1  0  0  0  0  0  0  0
4            1  0  0  1  0  0  0  0  0  0
5            1  0  0  0  1  0  0  0  0  0
6            1  1  0  0  1  0  0  0  0  0
7            1  0  1  0  1  0  0  0  0  0
8            1  0  0  1  1  0  0  0  0  0
9            1  0  0  0  0  1  0  0  0  0
10           1  1  0  0  0  1  0  0  0  0
11           1  0  1  0  0  1  0  0  0  0
12           1  0  0  1  0  1  0  0  0  0
13           1  0  0  0  0  0  1  0  0  0
14           1  1  0  0  0  0  1  0  0  0
15           1  0  1  0  0  0  1  0  0  0
16           1  0  0  1  0  0  1  0  0  0
17           1  0  0  0  0  0  0  1  0  0
18           1  1  0  0  0  0  0  1  0  0
19           1  0  1  0  0  0  0  1  0  0
20           1  0  0  1  0  0  0  1  0  0
21           1  0  0  0  1  0  0  1  0  0
22           1  1  0  0  1  0  0  1  0  0
23           1  0  1  0  1  0  0  1  0  0
24           1  0  0  1  1  0  0  1  0  0
25           1  0  0  0  0  1  0  1  0  0
26           1  1  0  0  0  1  0  1  0  0
27           1  0  1  0  0  1  0  1  0  0
28           1  0  0  1  0  1  0  1  0  0
29           1  0  0  0  0  0  1  1  0  0
30           1  1  0  0  0  0  1  1  0  0
31           1  0  1  0  0  0  1  1  0  0
32           1  0  0  1  0  0  1  1  0  0
33           1  0  0  0  0  0  0  0  1  0
34           1  1  0  0  0  0  0  0  1  0
35           1  0  1  0  0  0  0  0  1  0
36           1  0  0  1  0  0  0  0  1  0
37           1  0  0  0  1  0  0  0  1  0
38           1  1  0  0  1  0  0  0  1  0
39           1  0  1  0  1  0  0  0  1  0
40           1  0  0  1  1  0  0  0  1  0
41           1  0  0  0  0  1  0  0  1  0
42           1  1  0  0  0  1  0  0  1  0
43           1  0  1  0  0  1  0  0  1  0
44           1  0  0  1  0  1  0  0  1  0
45           1  0  0  0  0  0  1  0  1  0
46           1  1  0  0  0  0  1  0  1  0
47           1  0  1  0  0  0  1  0  1  0
48           1  0  0  1  0  0  1  0  1  0
49           1  0  0  0  0  0  0  0  0  1
50           1  1  0  0  0  0  0  0  0  1
51           1  0  1  0  0  0  0  0  0  1
52           1  0  0  1  0  0  0  0  0  1
53           1  0  0  0  1  0  0  0  0  1
54           1  1  0  0  1  0  0  0  0  1
55           1  0  1  0  1  0  0  0  0  1
56           1  0  0  1  1  0  0  0  0  1
57           1  0  0  0  0  1  0  0  0  1
58           1  1  0  0  0  1  0  0  0  1
59           1  0  1  0  0  1  0  0  0  1
60           1  0  0  1  0  1  0  0  0  1
61           1  0  0  0  0  0  1  0  0  1
62           1  1  0  0  0  0  1  0  0  1
63           1  0  1  0  0  0  1  0  0  1
64           1  0  0  1  0  0  1  0  0  1

Without an intercept:

> model.matrix(~ w + x + y - 1, df)
   wa wb wc wd xf xg xh yj yk yl
1   1  0  0  0  0  0  0  0  0  0
2   0  1  0  0  0  0  0  0  0  0
3   0  0  1  0  0  0  0  0  0  0
4   0  0  0  1  0  0  0  0  0  0
5   1  0  0  0  1  0  0  0  0  0
6   0  1  0  0  1  0  0  0  0  0
7   0  0  1  0  1  0  0  0  0  0
8   0  0  0  1  1  0  0  0  0  0
9   1  0  0  0  0  1  0  0  0  0
10  0  1  0  0  0  1  0  0  0  0
11  0  0  1  0  0  1  0  0  0  0
12  0  0  0  1  0  1  0  0  0  0
13  1  0  0  0  0  0  1  0  0  0
14  0  1  0  0  0  0  1  0  0  0
15  0  0  1  0  0  0  1  0  0  0
16  0  0  0  1  0  0  1  0  0  0
17  1  0  0  0  0  0  0  1  0  0
18  0  1  0  0  0  0  0  1  0  0
19  0  0  1  0  0  0  0  1  0  0
20  0  0  0  1  0  0  0  1  0  0
21  1  0  0  0  1  0  0  1  0  0
22  0  1  0  0  1  0  0  1  0  0
23  0  0  1  0  1  0  0  1  0  0
24  0  0  0  1  1  0  0  1  0  0
25  1  0  0  0  0  1  0  1  0  0
26  0  1  0  0  0  1  0  1  0  0
27  0  0  1  0  0  1  0  1  0  0
28  0  0  0  1  0  1  0  1  0  0
29  1  0  0  0  0  0  1  1  0  0
30  0  1  0  0  0  0  1  1  0  0
31  0  0  1  0  0  0  1  1  0  0
32  0  0  0  1  0  0  1  1  0  0
33  1  0  0  0  0  0  0  0  1  0
34  0  1  0  0  0  0  0  0  1  0
35  0  0  1  0  0  0  0  0  1  0
36  0  0  0  1  0  0  0  0  1  0
37  1  0  0  0  1  0  0  0  1  0
38  0  1  0  0  1  0  0  0  1  0
39  0  0  1  0  1  0  0  0  1  0
40  0  0  0  1  1  0  0  0  1  0
41  1  0  0  0  0  1  0  0  1  0
42  0  1  0  0  0  1  0  0  1  0
43  0  0  1  0  0  1  0  0  1  0
44  0  0  0  1  0  1  0  0  1  0
45  1  0  0  0  0  0  1  0  1  0
46  0  1  0  0  0  0  1  0  1  0
47  0  0  1  0  0  0  1  0  1  0
48  0  0  0  1  0  0  1  0  1  0
49  1  0  0  0  0  0  0  0  0  1
50  0  1  0  0  0  0  0  0  0  1
51  0  0  1  0  0  0  0  0  0  1
52  0  0  0  1  0  0  0  0  0  1
53  1  0  0  0  1  0  0  0  0  1
54  0  1  0  0  1  0  0  0  0  1
55  0  0  1  0  1  0  0  0  0  1
56  0  0  0  1  1  0  0  0  0  1
57  1  0  0  0  0  1  0  0  0  1
58  0  1  0  0  0  1  0  0  0  1
59  0  0  1  0  0  1  0  0  0  1
60  0  0  0  1  0  1  0  0  0  1
61  1  0  0  0  0  0  1  0  0  1
62  0  1  0  0  0  0  1  0  0  1
63  0  0  1  0  0  0  1  0  0  1
64  0  0  0  1  0  0  1  0  0  1

We can see that when we use an intercept, model.matrix uses dummy encoding with each variable w, x, and y being turned into 3 dummy variables, plus an intercept column. So there is a total of 10 degrees of freedom.

When we don't use an intercept, model.matrix creates 4 dummy variables for w and 3 dummy variables for x and y (and no intercept column). So the number of degrees of freedom is still 10.

Answer 3

I totally agree with @Matthew Drury and @Cameron Bieganek's analysis of perfect collinearity and degree of freedom.

However, I want to argue here that we do not need to avoid perfect collinearity if we are using methods such as gradient descends as our optimizer. (Update, I just realized that there are more situations where we do not need to avoid collinearity: For example, We do not need to avoid perfect collinearity if we do regression with regularizer either. Since it will add a identity matrix $\lambda I$ to the matrix($X^TX$) below invertible again.)

The reason why we might want to avoid perfect collinearity is when we are using linear regression and our loss function is MSE, we could solve the closed form solution, which involves the inverse of a matrix about $X$, and perfect collinearity would make this matrix non invertible. However, in practice, as the computation of inverse matrix is quite expensive $\mathcal{O}(n^3)$, we could use other faster method such as gradient descend to compute an approximate solution and this process does not involves the inverse of matrix. Thus, perfect collinearity could be tolerated here.(Maybe this is why they do not have a warning) So, we could either use:

1. one hot with intercept or
2. one hot without intercept or
3. dummy with intercept.

and they would generate quite similar result.

I run a regression using mpg dateset with sklearn, apply one hot or dummy to the "origin" feature which has three categories and the result is quite similar: (Please pay attention to how the residual and the coef are alike to each other, I marked parameters in front of enumerated category terms with red rectangles, with the others are the parameters of some continuous features. Detailed code could be seen here, sorry for the lack of comment and a large part of the code comes from tensorflow turorials.)

Relationship between the three results:

1. BTW, we could also observe that for case of one hot without intercept, the parameters in front of the last three one hot features are actually equals to the element-wise sum of intercept and the parameters in front of the last three one hot features in one hot with intercept, which can be explained by the perfect collinearity.

2. We can also notice that in the dummy with intercept case, the intercept is actually the parameter(-13.71778...) of the second categorical encoded term in the one-hot without intercept case. And the two parameter of the categorical encoded term in the dummy case is the difference between the corresponding parameters wrt the second term parameter, which is consistent with the interpretation of the parameter before categorical terms in econometrics: how much different each of the other categories makes to the output compare to the basic category.