What are appropriate methods for preparing categorical features for recurrent networks to ensure efficient backpropagation?

Question

Given a 1D sequential categorical input variable, e.g. [rainy, sunny, rainy, cloudy, cloudy], with a small domain {rain, sunny, cloudy}, what encoding methods (e.g. one-hot, dummy, binary) and what scaling methods (e.g. standardisation, min-max scaling) are appropriate specifically for use with RNNs such as LSTM and GRU given their logistic activation functions in comparison to other NNs which tend to use ReLU?

In Yann A. LeCun et al - Efficient BackProp Chapter in Neural Networks: Tricks of the Trade: Second Edition, pages 9–48, LeCunn states that preprocessing of data should be performed in such a way so as to ensure that the activations of the neurons have unit variance and mean zero (which is of course not possible for sigmoid activations) to ensure backprop is efficient, and presumably to ensure that signals to later layers lie approximately in the region [-1, 1] where in the gradient of the logistic functions is greatest.

I believe this is of particular significance to LSTM/GRU RNNs since if one uses one-hot encoding (or some other binary style encoding) the distribution of the responses will not be distributed with unit variance (sigmoid and tanh) as per LeCun's directions.

addendum - This has nothing to do with the convolution operation (as per the tagged duplicate) in the context of a categorical variable. This is a question about what are the appropriate encoding and scaling methods for GRU/LSTM RNNs which are likely specific to these types of networks due to the logistic type activation functions they use.

Answer 1

The answer is probably entity embeddings for categorical variables. The idea is to employ a strategy similar to word embeddings: put the categories into a lower-dimensional Euclidean space, and a neural network will sort out how to put the more-similar things closer together. This is done by feeding the one-hot data to a linear layer with identity activation that is trained in the usual way using back-prop. For neural networks at least, this tends to work better than the one-hot encoding on its own.

And of course the utility of embeddings in models with recurrent cells has been established for several years in the context of language models.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables"

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

Alternatively, there's nothing inherently wrong with just feeding one-hot data directly to the model (in some cases, even scaling is unnecessary). It might not be as good as entity embeddings, but it's common practice, a little bit simpler, and in some cases perfectly adequate.

---

The power of this method has more to do with the usefulness of embeddings and less to do with the importance of scaling or distribution assumptions. You can put one-hot encoded data through recurrent units and the model will be adequate; this is common for character-level embeddings with language models.