6
$\begingroup$

I am facing an ML task with various categorical variables. Some examples include the following:

  • Binary variables (0,1).
  • Multilevel factors that can be ordered (low, medium, high).
  • Multilevel factors that cannot be ordered (red, blue, green).

I'm going to use a deep neural network to perform my classification task: however, I'd like to find a possibly comprehensive guide to encoding categorical features according to the most commonly used methods and current best practices. Is there a reference text?

I'm asking this since I usually need to do similar encoding tasks and I usually apply some rough fix, which is probably suboptimal, but that works more or less fine. This approach is, however, not an option in some cases and I would like to improve the way I handle such encoding tasks.

$\endgroup$
  • 1
    $\begingroup$ A good question, I would in fact like to see a good reference for this too. Sidenote: with a model this powerful you might be able to use all features just in their raw form, but I understand that this is not what you aim for. Additionally you might a) one-hot encode unordered factors, b) convert multilevel factors to plain numerics (this introduces an additional and probably wrong relation) or also one-hot encode them, then c) treat all those like continuous variables and apply scaling etc. before handing them to models. But I guess this is what you'd call a "rough fix", right? $\endgroup$ – geekoverdose Jul 26 '16 at 13:03
  • $\begingroup$ @geekoverdose Hi, thank you for your comment and suggestions. I think I will try most of them. Yes, I agree with you, that is a "rough fix", however that seems also to be "industry standards", I understand that other solutions might be too much specific... However I think the point of the question is to have a "general idea" of what solution could be the first try in order not to wander around all day for the "best" solution. One hot encodings seems to be the preferred choice IMHO. $\endgroup$ – mickkk Jul 26 '16 at 17:06
10
$\begingroup$

Binary variables

No encoding is needed: use them as is.

Nominal data

When you have an variable that can take on a finite number of values, that's called a categorical variable. When the values can't be ordered (e.g., red, blue, green), that's called a nominal variable. A nominal variable is one kind of categorical variable.

For nominal variables, the usual way to encode them is with a one-hot encoding. If there are $N$ possible values for the variable, you map each value to a $N$-vector that has a $1$ in the position corresponding to that value and $0$ elsewhere.

For instance: red $\mapsto (1,0,0)$, blue $\mapsto (0,1,0)$, green $\mapsto (0,0,1)$.

Ordinal data

When you have a categorical variable where the values can be ordered (sorted), but the ordering doesn't imply anything about how much they differ, that's called a ordinal variable (see ordinal data).

For example, suppose you have a ranking: John finished in 3rd place, Jane in 6th place. You know that John finished before Jane, but that doesn't necessarily mean that John was $6/3=2$ times as fast as Jane.

You can encode ordinal data using the thermometer trick. If there are $N$ possible values for the variable, then you map each value to a $N$-vector, where you put a $1$ in the position that matches the value of the variable and all subsequent position.

For instance: first place $\mapsto (1,1,1)$, second place $\mapsto (0,1,1)$, third place $\mapsto (0,0,1)$.

You can also apply binning if $N$ is too large, but usually it's better not to do that.

Numerical variables

Finally, you may encounter variables that directly measure a number, and where they can be not only ordered, but also subtracted or divided. Then, it's typically best to use the number directly, or possibly use the logarithm of the number. (You might take the logarithm if the number represents a ratio, or if there is a very wide range of values.)

Useful background

To understand these terms, it's helpful to learn about "level of measurements": https://en.wikipedia.org/wiki/Level_of_measurement.

Scaling

Finally, when you're using neural networks or "deep learning", you'll normally want to standardize/rescale all numerical attributes before applying deep learning. I suggest you treat that as a separate process from the feature mappings mentioned above, to be performed after you apply the feature mapping.

$\endgroup$
  • $\begingroup$ @DW Would that create a large number of features/columns? Say I have 100 ordinal variables and each takes on 5 possible values, the thermometer encoding would yield 500 features. $\endgroup$ – horaceT Sep 29 '16 at 23:37
  • 1
    $\begingroup$ @horaceT, yes, it will. Usually, that's not a big deal. Good ML algorithms should be able to handle it, especially if you use some kind of regularization. If yours can't, use feature selection, or use a different ML algorithm; if neither of those work, you'll have to fall back to something custom. here's no one-size-fits-all answer that will work in all situations, but these are standard techniques that often work -- try them first, before looking for something fancier. $\endgroup$ – D.W. Sep 30 '16 at 2:57
  • $\begingroup$ For nominal data should we create N or N-1 different values for the encoding (i.e. one-hot vs dummy)? $\endgroup$ – user90772 Jul 8 '17 at 23:39
  • $\begingroup$ @user90772, sounds worth asking as a separate question, after searching to see if it is answered elsewhere. Make sure to mention in your question what you're doing with the variables (classification? regression? using what ML method/technique?) $\endgroup$ – D.W. Jul 9 '17 at 4:04
  • $\begingroup$ Could you please comment here? stats.stackexchange.com/questions/290526/… $\endgroup$ – user90772 Jul 9 '17 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.