2
$\begingroup$

we know that for non-numeric categorical features, such as country, color, they can be transform into numeric value by one-hot encoding.

For instance, suppose there are 7 color options for a car, red will then be encoded as [0 0 0 1 0 0 0]. We can therefore include these 7 numeric values into a logistic regression model.

However, one of linear regression's assumptions is that, predictors are independent. One hot encoding obviously violates this assumption.

Probably I miss understood something here. Any thoughts?

PS: as answered by Dave and Gordon below: the trick is to use 6, instead of 7, new features. None of these 6 could be a linear combination of the other 5.

Improve this question
$\endgroup$
5
  • $\begingroup$ How does "logistic regression" enter in to this question? Are you treating the 0/1 coded values as x-variables or as the y-variable? If they are predictors (x-variables) then you seem to be doing standard anova rather than logistic regression. If they are the y-variable, then you would indeed have an independence problem. $\endgroup$ – Gordon Smyth Sep 29 '19 at 8:26
  • $\begingroup$ the color is a predictor, i.e., it is part of X, not y. $\endgroup$ – eight3 Sep 30 '19 at 7:12
  • $\begingroup$ Then this a standard encoding that has been used in statistics for 100 years. You avoid any collinearity problem by inserting 6 numeric values into the regression model, not 7. The color level left out becomes the reference level and is absorbed into the intercept term. If you did incorrectly insert all 7 variables into the regression model, then most regression software tools will just remove the last one automatically to avoid the problem. $\endgroup$ – Gordon Smyth Oct 1 '19 at 2:25
  • $\begingroup$ If you are using R, just include +color in the linear model formula and the correct number of coding variables will be created automatically. You never need to create the coding variables yourself. $\endgroup$ – Gordon Smyth Oct 1 '19 at 8:25
  • $\begingroup$ @GordonSmyth thanks! $\endgroup$ – eight3 Oct 1 '19 at 8:29
3
$\begingroup$

There is no such assumption about predictors. This is an easy mistake to make, because of the Gauss-Markov theorem requiring uncorrelated errors. Another way to make this mistake is by confusing independence of predictors and independence of predictions.

I've made this mistake. Probably most of us have made this mistake. Welcome to the club of learning that it's false. :)

I do want to comment on using all seven colors in the model. The typical way to do it would be to include six color variables. The seventh gets sucked into the intercept term. Not doing it this way means that your model matrix can add up all of the color columns and get the intercept column of 1s, which makes your model matrix not have full rank.

Red: [1 0 0 0 0 0]$^T$

Orange: [0 1 0 0 0 0]$^T$

Yellow: [0 0 1 0 0 0]$^T$

Green: [0 0 0 1 0 0]$^T$

Blue: [0 0 0 0 1 0]$^T$

Indigo: [0 0 0 0 0 1]$^T$

Violet: [0 0 0 0 0 0]$^T$

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the answer. However, as per statisticssolutions.com/assumptions-of-linear-regression . multicollinearity surely violates the assumption of linear regression. Do you mean 'dependency' is not necessarily co-linearity? $\endgroup$ – eight3 Sep 30 '19 at 7:09
  • $\begingroup$ It depends on what you're doing. Multicollinearity will affect parameter inference. However, if you only include those seven colors in your regression (e.g. ANOVA), there is no issue of multicollinearity. No linear combination of six can given the seventh. $\endgroup$ – Dave Sep 30 '19 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.