# Should I scale and normalize data when training character based LSTM?

I'm working on a sequence classifier that classifies sequences of characters. There are 100 classes. Simple solution I did is just assign different int value to each character and then train LSTM with that. So for example, 'a' becomes '1', 'b' becomes '2' etc. It works really good so far on my dummy set, but I'm wondering if there is any benefit in scaling those values to interval [0, 1] and normalization?

• do you use any character embeddings (i.e., using mapping each character to a vector instead of mapping to an integer)? – Franck Dernoncourt Nov 23 '16 at 15:55
• @FranckDernoncourt - I don't use mapping for characters because I didn't see the point since characters don't have much meaning by themselves like words do. Why do you ask? – Drag0 Nov 23 '16 at 20:44
• Also, I use LSTM to recognize programming languages so I removed natural-language tag. Hope it's ok with you @FranckDernoncourt :) – Drag0 Nov 23 '16 at 20:45
• mapping each character to a integer introduces an ordering (1 < 2 < 3 etc.), so people typically map each character to a vector instead. Thanks for correcting the tag. – Franck Dernoncourt Nov 24 '16 at 7:12
• @FranckDernoncourt - ok, thanks. I will map chars to vectors and see what happens. – Drag0 Nov 24 '16 at 9:05

To directly answer your question: no you should not scale your input values, which are categorical in nature. You will use something like ' '=1, 'a'=2, 'b'=3, ... as input, and at some point this will be turned into a one-hot encoding -- whether by you or the software you are using. (Depending on the software you are using, 0 may have special meaning, so I start with 1.)

This one-hot encoding is very sparse by definition. An embedding (https://en.wikipedia.org/wiki/Word_embedding) is commonly used to turn sparse, discrete encodings into dense, continuous encodings where relationships among the vocabulary are reflected in spatial relationships. An embedding will be lower-dimensional than your vocabulary -- that's part of what makes it denser. These embeddings have historically been for words, but the same idea applies to whatever your input, which in your case is individual characters.

I personally don't believe you need to embed your dozens of letters into hundreds of dimensions. Word2vec encodings of millions of words are often only 300 dimensions, and there is a lot of information in the words to encode. (Embedding in a higher dimension than your one-hot encoding would be counter-productive as far as I can tell.)

EDIT: My context -- given as an example, below -- is a sequence-to-sequence model that has one output for each input. Some sequence-to-sequence models, such as language translation, may give one output for several inputs or several outputs for one input, and they use an encoder-decoder architecture.

EDIT: It turns out that the output of the encoder is also called an embedding. Not in the Word2Vec sense of input items (words or characters), but in the sense of summarizing the entire input. In this use case, the "embedding" must be large, since it is the single piece of information passed from the encoder to the decoder and has to summarize the input well. It's all the decoder has to work with.

EDIT: I believe this is where @Sycorax and I have been miscommunicating.

RNNs will have hundreds or thousands of LSTM (or GRU, etc) units in a layer, but don't confuse that with the dimensionality of the input -- i.e. the word/character embedding. These recurrent layers are very wide because there's a lot of stuff to remember and that's what LSTMs do. If you don't have enough LSTMs (or GRUs, etc) in a layer to remember what you need to remember as you move through the text, your model won't do well.

See the excellent page recommended by @Sycorax (http://karpathy.github.io/2015/05/21/rnn-effectiveness ) for some cool insights into what an RNN might be trying to remember. How much your model has to "keep in mind" at any point is somewhat the tradeoff of using character-level RNNs: your vocabulary is much smaller and of fixed size but the model has a lot more things it has to implicitly keep in mind.

As an example, I am working on a character-based, sequence-to-sequence RNN, using LSTM nodes. I only have several hundred training documents, which is a handicap. I used Keras' embedding layer to embed my 90-character vocabulary into 2 dimensions. (So a 90-wide, sparse, one-hot matrix is turned into a 2-wide, dense, continuous-valued matrix. One row per character.)

When I plot each character by their embedding, the two dimensions are highly correlated, which I believe means that there is not a full two-dimensions-worth of information in my data/task. I can see that numbers have clustered at one end of the plot, which is nice, but other punctuation and letters are jumbled along the diagonal. (Perhaps they will never be grouped, based on my task, or perhaps groupings won't make sense to humans, but my current theory is that there's not much information -- that will solve my task -- in different punctuation or letters.)

Just some anecdotal evidence, but I thought I'd toss it in.

• +1 This is a good answer -- I think you have the right of it, that we want to have a lower-dimensional representation of the words/characters. – Sycorax Aug 19 '18 at 4:30