I'm never sure when to use one-hot encoding for non-ordered categorical variables and when not to. I use it whenever the algorithm uses a distance metric to compute similarity. Can anyone give a general rule of thumb as to what types of algorithms would require non-ordered categorical features to be one-hot-encoded and which ones wouldn't?
$\begingroup$
$\endgroup$
6
-
2$\begingroup$ Are you asking about the need to code categorical data into some type of "contrast variables" or specifically into dummy (one-hot) type? $\endgroup$– ttnphnsCommented Jun 30, 2017 at 14:03
-
1$\begingroup$ This question is somewhat broad, but a simple answer that addresses the nature of OHE can clear up the OP's confusion. The existence of such answers implies this question is answerable. I'm voting to leave open. $\endgroup$– gung - Reinstate MonicaCommented Jun 30, 2017 at 15:18
-
$\begingroup$ @ttnphns honestly I don't know what you mean by contrast variable. I'm only familiar with dummy. $\endgroup$– cosmosaCommented Jun 30, 2017 at 20:53
-
$\begingroup$ Cosmos, dummy (=indicator =one-hot) is just one of a number of ways to encode categorical cariables in analyses. Theses ways unitely are called "contrast variables". See stats.meta.stackexchange.com/q/4669/3277 and stats.stackexchange.com/a/221868/3277 $\endgroup$– ttnphnsCommented Jul 1, 2017 at 12:29
-
1$\begingroup$ I think the complete list you ask for is going to be hard to produce. $\endgroup$– mdeweyCommented Jul 1, 2017 at 13:05
|
Show 1 more comment
3 Answers
$\begingroup$
$\endgroup$
1
Most algorithms (linear regression, logistic regression, neural network, support vector machine, etc.) require some sort of the encoding on categorical variables. This is because most algorithms only take numerical values as inputs.
Algorithms that do not require an encoding are algorithms that can directly deal with joint discrete distributions such as Markov chain / Naive Bayes / Bayesian network, tree based, etc.
Additional comments:
One hot encoding is one of the encoding methods. Here is a good resource for categorical variable encoding (not limited to R). R LIBRARY CONTRAST CODING SYSTEMS FOR CATEGORICAL VARIABLES
Even without encoding, distance between data points with discrete variables can be defined, such as hamming distance or Levenshtein Distance
answered Jun 30, 2017 at 14:15
-
$\begingroup$ It is still not clear if one hot encoding is required for MOST ALGORITHMS. You just say that encoding is required. But is it one hot encoding? $\endgroup$ Commented Apr 5, 2019 at 4:23
$\begingroup$
$\endgroup$
2
Can anyone give a list of what algorithms would require categorical features to be one-hot-encoded and which ones wouldn't?
AFAIU, it has to do more with the particular data, less with the particular algorithm. Specifically, it depends on whether there is some meaningful order in the categories or not.
Consider two cases. In the first you have the categories bad, meh, good, and in the second you have apple, orange, pear. There is a natural order in the first case, because meh is probably in between bad and good, but probably nothing similar happens in apple, orange, pear.
If you avoid one-hot encoding for the first case, you're "losing" the information about the order. If you use one-hot encoding for the second case, you're assigning some order to the categories that is not naturally true.
I do it whenever the algorithm uses a distance metric to compute similarity.
Why? Suppose one of the features is a categorical bad, meh, good, and you have three instances, 1, 2, and 3, where they are identical, except that 1 is bad, 2 is meh, and 3 is good. You probably want to express to the algorithm that 1 is more similar to 2 than it is to 3.
-
2$\begingroup$ That's a good answer. I should have clarified the question though to include non-ordered categorical variables as well. In that case, it should always be one hot encoded? $\endgroup$– cosmosaCommented Jun 30, 2017 at 10:58
-
$\begingroup$ @cosmos1990 IMHO, as a rule of thumb, for non-ordered categorical data, one-hot encoding is the way to go (as opposed to assignment of numerical values). $\endgroup$ Commented Jun 30, 2017 at 11:05
$\begingroup$
$\endgroup$
No machine learning algorithm requires one hot encoding. It is one method for dealing with categorical variables. Dummy variables is another. Traditionally, dummy variables was the preferred solution. For example, the R function lm() automatically creates dummy variables for categorical data. If you are using python and scikt-learn then I believe many of it's algos require one-hot encoding of categorical variables. I believe that tensorFlow also requires one-hot encoding. These are choices of how the variable is encoded. There is no reason why dummy variables couldn't be used in the code instead. This all has to deal with the actual code implementation of the algorithm.
As hxd1011 points out the issue of describing the 'distance' between categorical variables is a delicate issue. In addition to the distances mentioned there is also Jaccard distance. Some ML methods, particularly SVM's are inappropriate for categorical data and adding categorical variables can/will (either, both, you decide) lead to models with very poor predictive power. Most ensemble models handle categorical data 'as is' and require no pre-processing.
