3
$\begingroup$

The Tensorflow LSTM example uses a word embedding layer (https://www.tensorflow.org/versions/r0.11/tutorials/recurrent/index.html). However, it mentions that 'the embedding matrix will be initialized randomly and the model will learn to differentiate the meaning of words just by looking at the data'. I think this means that pretrained embeddings (e.g. using Word2Vec) are not used. I have two questions with regards to this.

  1. Are there any variables/weights (e.g. adjusted during back prop) associates with the word_embedding layer in this RNN? Or is it purely a static lookup table to randomly assigned variables?

  2. Do random word embeddings have any advantages over just adding a regular hidden layer with the same dimensions as the randomly initialized word embedding layer?

Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I think this means that pretrained embeddings (e.g. using Word2Vec) are not used.

Correct

Are there any variables/weights (e.g. adjusted during back prop) associated with the word_embedding layer in this RNN? Or is it purely a static lookup table to randomly assigned variables?

First option: there are any variables/weights (e.g. adjusted during back prop) associated with the word_embedding layer in this RNN.

Do random word embeddings have any advantages over just adding a regular hidden layer with the same dimensions as the randomly initialized word embedding layer?

It is the equivalent: adding a random word embedding layer means adding a regular hidden layer.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you very much. tf.trainable_variables() does indeed show that the embeddings variable is trainable. When you say it's equivalent, does that mean behind the scenes tf.nn.embedding_lookup is creating activation(Wx+B) tensors on each dimension of the vector automatically? $\endgroup$
    – user136335
    Commented Oct 26, 2016 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.