12
$\begingroup$

I'm never sure when to use one-hot encoding for non-ordered categorical variables and when not to. I use it whenever the algorithm uses a distance metric to compute similarity. Can anyone give a general rule of thumb as to what types of algorithms would require non-ordered categorical features to be one-hot-encoded and which ones wouldn't?

$\endgroup$
  • 2
    $\begingroup$ Are you asking about the need to code categorical data into some type of "contrast variables" or specifically into dummy (one-hot) type? $\endgroup$ – ttnphns Jun 30 '17 at 14:03
  • 1
    $\begingroup$ This question is somewhat broad, but a simple answer that addresses the nature of OHE can clear up the OP's confusion. The existence of such answers implies this question is answerable. I'm voting to leave open. $\endgroup$ – gung - Reinstate Monica Jun 30 '17 at 15:18
  • $\begingroup$ @ttnphns honestly I don't know what you mean by contrast variable. I'm only familiar with dummy. $\endgroup$ – cosmosa Jun 30 '17 at 20:53
  • $\begingroup$ Cosmos, dummy (=indicator =one-hot) is just one of a number of ways to encode categorical cariables in analyses. Theses ways unitely are called "contrast variables". See stats.meta.stackexchange.com/q/4669/3277 and stats.stackexchange.com/a/221868/3277 $\endgroup$ – ttnphns Jul 1 '17 at 12:29
  • 1
    $\begingroup$ I think the complete list you ask for is going to be hard to produce. $\endgroup$ – mdewey Jul 1 '17 at 13:05
6
$\begingroup$

Most algorithms (linear regression, logistic regression, neural network, support vector machine, etc.) require some sort of the encoding on categorical variables. This is because most algorithms only take numerical values as inputs.

Algorithms that do not require an encoding are algorithms that can directly deal with joint discrete distributions such as Markov chain / Naive Bayes / Bayesian network, tree based, etc.

Additional comments:

$\endgroup$
  • $\begingroup$ It is still not clear if one hot encoding is required for MOST ALGORITHMS. You just say that encoding is required. But is it one hot encoding? $\endgroup$ – Prometheus Apr 5 at 4:23
4
$\begingroup$

Can anyone give a list of what algorithms would require categorical features to be one-hot-encoded and which ones wouldn't?

AFAIU, it has to do more with the particular data, less with the particular algorithm. Specifically, it depends on whether there is some meaningful order in the categories or not.

Consider two cases. In the first you have the categories bad, meh, good, and in the second you have apple, orange, pear. There is a natural order in the first case, because meh is probably in between bad and good, but probably nothing similar happens in apple, orange, pear.

If you avoid one-hot encoding for the first case, you're "losing" the information about the order. If you use one-hot encoding for the second case, you're assigning some order to the categories that is not naturally true.

I do it whenever the algorithm uses a distance metric to compute similarity.

Why? Suppose one of the features is a categorical bad, meh, good, and you have three instances, 1, 2, and 3, where they are identical, except that 1 is bad, 2 is meh, and 3 is good. You probably want to express to the algorithm that 1 is more similar to 2 than it is to 3.

$\endgroup$
  • 2
    $\begingroup$ That's a good answer. I should have clarified the question though to include non-ordered categorical variables as well. In that case, it should always be one hot encoded? $\endgroup$ – cosmosa Jun 30 '17 at 10:58
  • $\begingroup$ @cosmos1990 IMHO, as a rule of thumb, for non-ordered categorical data, one-hot encoding is the way to go (as opposed to assignment of numerical values). $\endgroup$ – Ami Tavory Jun 30 '17 at 11:05
2
$\begingroup$

No machine learning algorithm requires one hot encoding. It is one method for dealing with categorical variables. Dummy variables is another. Traditionally, dummy variables was the preferred solution. For example, the R function lm() automatically creates dummy variables for categorical data. If you are using python and scikt-learn then I believe many of it's algos require one-hot encoding of categorical variables. I believe that tensorFlow also requires one-hot encoding. These are choices of how the variable is encoded. There is no reason why dummy variables couldn't be used in the code instead. This all has to deal with the actual code implementation of the algorithm.

As hxd1011 points out the issue of describing the 'distance' between categorical variables is a delicate issue. In addition to the distances mentioned there is also Jaccard distance. Some ML methods, particularly SVM's are inappropriate for categorical data and adding categorical variables can/will (either, both, you decide) lead to models with very poor predictive power. Most ensemble models handle categorical data 'as is' and require no pre-processing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.