Dictionary creation in deep learning for NLP

Question

After watching the NAACL2013 videos and papers, I have some doubts on how a RNN works.

All the words are represented in a matrix $L\in M_{k \times |V|}$ where $k$ is an arbitrary number of dimensions and $V$ the vocabulary.

How is this vocabulary created?

Let's say that I train my network with some text. At this point I have a vocabulary of the unique words occurring in this text. What happens if I give to the RNN a word never met during training as input? Will it be ignored or a new column is added to $L$?

New words

In case new words are added to $L$, what should I expect when, for instance, plotting word similarities? Let's say that in my taining dataset I have France and Germany, then I give as input a sentence containing Spain. Will the RNN put it close to other UE states or only close to nouns?

Sentences of variable length

In text, a sentence has a very variable number of words, but the number of units in the input layer is fixed. Is splitting sentences in blocks of $|input\_units|$ length enough?

Answer 1

How is this vocabulary created?

Since you use 1-out-of-K to encode words to sparse vectors, you cannot input a word outside your vocabulary. To handle such cases, it is common to initialise you vocabulary with an open-source dictionary you can find online.

New words

By adding a new column in L you need to retrain your whole network as the weight matrices $W_enc$ for encoding have changed from $N_L \times N_{emb}$ to $N_L +1 \times N_{emb}$. It is also possible to fine-tune instead of full retraining.

In terms of conceptual explanation of where the vector would be close, it is impossible to translate it exactly to human terms as the embeddings are just numbers. However, you would still need to retrain your network for the new column in the matrices to come to any conclusions.

Sentences of variable length

In case you use RNNs, you will not have a problem as you input one word at a time, and backpropagate through a defined number of steps (e.x. 20 words). Therefore, you can either limit the backpropagation or use a special padding word (e.x. "000000") to fill up the missing words. Given that you use LSTMs, it will learn to ignore it.

If you use Conv Nets, then you definitely need padding to fill the missing words, while the input size is defined by your largest sentence.