4
$\begingroup$

I am using a set of pretrained word embeddings (GloVe) in my NN model. So far i just omit OOV words and i add zero-padding at the end of a sentence to have a fixed length for all the sentences/docs (for CNNs).

I want to add special padding tokens for the beggining/end of a sentence and for OOV words (START,END,UNK respectively). My problem is what values should the embeddings of these tokens have. For the UNK i was thinking to randomly initialize it from a uniform distribution (-0.5, 0.5).

np.random.uniform(low=-0.5, high=0.5, size=(10,))
array([ 0.48440237,  0.43741531, -0.46003191,  0.26630407, -0.26999162,
        0.48189692,  0.16413969, -0.3330607 ,  0.45768477,  0.14461057])

I am not even sure if that is correct. Also what abbout START,END?

I should note that my the embeddings are "frozen" / will not be trained any further for the task, so the values i will pick for the padding tokes will not change.

Is there a way to pick some sensible values?

$\endgroup$
6
  • $\begingroup$ Why freeze the embeddings, and why add START/END tokens? $\endgroup$ – Franck Dernoncourt Nov 30 '16 at 3:51
  • 1
    $\begingroup$ @FranckDernoncourt I do sentiment analysis so i will try to answer in this context. I freeze the embeddings for 2 reasons (if you are intrested in this you may want to take a look at this youtube.com/…): 1) i drastrically reduce the parameters of the model (faster training, avoid overfitting) ... $\endgroup$ – Christos Baziotis Nov 30 '16 at 14:56
  • 1
    $\begingroup$ continue > 2) the words that exist in the training set will be moved in the embedding space (their embeddings wil change) to reflect their sentiment, but those words in the test set that are not in the training set will remain at their original position. So the model will treat the words that occupy a certain reagion (more likely regions) in the embedding space as positive/negative. And that means that the words that were not seen during training will remain in region in space that probably does not reflect their "true" sentiment. $\endgroup$ – Christos Baziotis Nov 30 '16 at 14:56
  • 1
    $\begingroup$ continue > Regarding the START/END tokens: a word may have different importance or meaning if it is at the beggining of a sentence or before the end. In the case of RNNs i have read (i will try to find a ref) that adding these special tokes helps the model learn this differencies. $\endgroup$ – Christos Baziotis Nov 30 '16 at 15:00
  • $\begingroup$ Thanks, interesting, yes if you find the reference I am interested: Does adding START/END tokens improved classification performances when doing sentiment analysis with neural networks? $\endgroup$ – Franck Dernoncourt Nov 30 '16 at 18:27
1
$\begingroup$

For CNN architecture usually the input is (for each sentence) a vector of embedded words [GloVe($w_0$), GloVe($w_1$), ..., GloVe($w_N$)] of fixed length N. So commonly you preset the maximum sentence length and just pad the tail of the input vector with zero vectors (just as you did). Marking the beginning and the end of the sentence is more relevant for the RNN, where the sentence length is expected to be variable and processing is done sequentially.

Having said that, you can add a new dimension to the GloVe vector, setting it to zero for normal words, and arbitrarily to (say) 0.5 for BEGIN, and 1 for END and a random negative value for OOV.

As for unknown words, you should ;) encounter them very rear, otherwise you might consider training the embeddings by yourself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.