2
$\begingroup$

Suppose you have linear model and a single feature named "color" (for the sake of simplicity). In linear model you look for a coefficient $\theta_1$ which is going to multiply this feature $x$ in your hypothesis function $h\left(x\right) = \theta_1x$ + $\theta_2$. Likewise if you had something like neural network or logistic regression you would look for a coefficient $\theta_1$ which is going to multiply this color featue in the hypothesis function $h\left(x\right) = \mathrm{sigm}(\theta_1x$ + $\theta_2)$.

So if your colors are encoded using numbers $1$ and $2$, then it doesn't make sense if the red color results in $\theta \cdot 1$ and the blue color results in $\theta \cdot 2$ whatever that $\theta$ is.

My question: Is one hot encoding preferable only in such models where you multliply the feature by some coefficient? For example does it matter which encoding to use in random forest? (I'm not sure but as I know when you calculate entropy you don't multiply features by coefficients in the way shown above)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

One-hot encoding ensures that no implicit order is imposed on the feature while integer/label encoding benefits from it. If there is no inherent ordering, the usual approach is one-hot encoding, however sometimes (e.g. in high cardinality) other options can be preferred.

How you encode your features always matters because it changes the model's behavior. For example, in random forests, or simply decision trees, with label encoder it's possible to split the samples into two categories where one side is say Red, Blue and the other side is Green, Yellow if they are ordered as Red, Blue, Green, Yellow (i.e. 1,2,3,4), by splitting wrt value $2.5$. This is not possible in one split with one-hot encoding. However, it may or may not make sense doing this in the context of the problem. This naturally affects the branching of your tree(s) because of hyper-parameters like max depth.

Therefore, we can't say that OHE only matters for models where you multiply your features with coefficients.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thank you for the reply! But it doesn't seem to me that you've answered my question. I know the reasons to use OHE. I asked: "Do only models, where you multiply the feature by some coefficient, benefit most from OHE?" $\endgroup$
    – mathgeek
    Commented Oct 7, 2021 at 7:59
  • $\begingroup$ I wrote the first paragraph just to remind the difference between one-hot encoding and the label encoding for the general audience. I tried to address your question in the second paragraph, but I think I need to explain more because I thought I addressed it. In short, benefits or not, it matters how you encode your features, and this is not constrained to models where you multiply features with coefs. $\endgroup$
    – gunes
    Commented Oct 7, 2021 at 8:06
  • $\begingroup$ Ah, now I got it. I just didn't see it that way. I have my last confusion. If we are only constrained with, say, linear models, can we always use OHE, no matter what categorical variable, ordinal or nominal, we have? My line of reasoning is the following one. Suppose you have an ordinal variable. When you label encode it, you impose that "blue" => 2 is twice as big as "red" => 1. Whereas it can be that "blue" should be 1.7 * "red". And if you one-hot-encode it, you just let your linear model to choose this coefficient (2 or 1.7 or smth else) on its own. Is that right? $\endgroup$
    – mathgeek
    Commented Oct 7, 2021 at 8:23
  • $\begingroup$ I mean, assuming that the curse of dimensionality and memory consumption are not the issue. $\endgroup$
    – mathgeek
    Commented Oct 7, 2021 at 8:37
  • $\begingroup$ Increasing number of weights indirectly affects the regularization loss, and it's hard to guess what will happen. In addition, plain OHE causes dummy variable trap and makes the matrix singular when there is bias (since the sum of all the features will be always 1). So, you'll need to decrease the num of vars by 1. You don't know which function of a feature is truly related to the target variable. We sometimes add $x^2, x^3$ type of polynomial features to the linear models. How would you know the ordering you impose, a feature as a function $x$, $i.e. f(x)$, is good or bad without validation? $\endgroup$
    – gunes
    Commented Oct 8, 2021 at 6:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.