How to recode categorical variable into numerical variable when using SVM or Neural Network

Question

To use SVM or Neural Network it needs to transform (encode) categorical variables into numeric variables, the normal method in this case is to use 0-1 binary values with the k-th categorical value transformed to be (0,0,...,1,0,...0) (1 is on the k-th position). Is there other methods to do this, especially when there are a large number of categorical values(e.g.10000) such that the 0-1 representation will introduce a large number of additional dimensions(input units) in Neural Network which seems not quite desired or expected?

I am asking about general strategies.

Answer 1

In NLP, where words are typically encoded as 1-of-k, the use of word embeddings has emerged recently. The wikipedia page with its references is a good start.

The general idea is to learn a vectorial representation $x_i \in \mathbb{R}^n$ for each word $i$ where semantically similar words are close in that space. Consequently, the inputs are of size $n$ instead of the size of the vocabulary.

Maybe you can transfer that idea to your setting.

Answer 2

The 'standard' methods are: one-hot encoding (which you mentioned in the question). If there are too many possible categories, but you need 0-1 encoding, you can use hashing trick.

The other frequently used method is averaging answer over category: see picture from comment at kaggle.

Answer 3

You can use dummyVars in R, from the caret package. It will automatically create different columns based on number of levels. Afterwards, you can use cbind and attach it to you original data. Other options include model.matrix and sparse.model.matrix.

Answer 4

You can try binary encoding which is more compact and sometimes outperforms one-hot. You can implement categorical embedding in Keras, for example.

Answer 5

You can use entity encoding, which is a more sophisticated network structure. It adds between 1 and $k-1$ hidden, linear neurons between the categorical input and the first fully-connected layer. This has some nice empirical results behind it.

"Entity Embeddings of Categorical Variables" by Cheng Guo, Felix Berkhahn

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.