1
$\begingroup$

Often it is best to standardize data before inputting it into a machine learning algorithm. This is also the case with deep learning algorithms such as convolutional neural network. However, when looking into the code on sentiment analysis from the tutorial on recurrent neural networks (RNN) / long short-term memory networks (LSTM) at deeplearning.net, I've found that the input isn't standardized. One example of an input to the algorithm (a sentence) is below:

X = [17, 25, 769, 83, 3, 14, 80, 62, 3221, 5, 928, 3, 1782, 6, 1, 1, 771, 24, 3350, 7, 1112, 228, 5, 3978, 4, 17, 25, 1212, 80, 6, 189, 7, 62, 1293, 5, 514, 4, 2, 131, 10, 1146, 480, 59, 413, 213, 117, 3, 14, 824, 69, 611, 2, 239, 73, 222, 72, 2338, 2, 67, 147, 4, 15, 1164, 123, 17, 10, 6, 100, 111, 23, 45, 228, 25, 4427, 8, 1131, 73, 31, 4]
y = [1]

These input values differ greatly from the standard of having $\mu = 0$ and $\sigma = 1$. How can it be that the input is not standardized? Would it not help to standardize it, and why is the RNN/LSTM then different from other neural networks such as CNN?

Edit: Perhaps it might be due to use of Adadelta as a way of training the network? It does have adaptive learning rates, but I don't know enough about it to say whether that is the cause.

$\endgroup$
1
$\begingroup$

The model in the deeplearning.net tutorial learns an embedding of the words, which maps from an integer word index (your X above) to a scalar vector for each word. These embedded words are the input to the LSTM.

In lstm.py, take a look at the build_model() function, which has the following line:

emb = tparams['Wemb'][x.flatten()].reshape([n_timesteps,
                                            n_samples,
                                            options['dim_proj']])

Here the model parameter Wemb is the embedding, and the variable x contains the integer word indexes from your data. The result is the scalar vectors emb, one vector of dim_proj scalars for each word. Hopefully it is clear what is going on with flatten() and reshape(). The word embedding approach is explained in Recurrent Neural Networks with Word Embeddings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.