11
$\begingroup$

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

$\endgroup$
18
$\begingroup$

Embedding layers in Keras are trained just like any other layer in your network architecture: they are tuned to minimize the loss function by using the selected optimization method. The major difference with other layers, is that their output is not a mathematical function of the input. Instead the input to the layer is used to index a table with the embedding vectors [1]. However, the underlying automatic differentiation engine has no problem to optimize these vectors to minimize the loss function...

So, you cannot say that the Embedding layer in Keras is doing the same as word2vec [2]. Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words. With Keras's embedding layer, you are just trying to minimize the loss function, so if for instance you are working with a sentiment classification problem, the learned embedding will probably not capture complete word semantics but just their emotional polarity...

For example, the following image taken from [3] shows the embedding of three sentences with a Keras Embedding layer trained from scratch as part of a supervised network designed to detect clickbait headlines (left) and pre-trained word2vec embeddings (right). As you can see, word2vec embeddings reflect the semantic similarity between phrases b) and c). Conversely, the embeddings generated by Keras's Embedding layer might be useful for classification, but do not capture the semantical similarity of b) and c).

enter image description here

This explains why when you have a limited amount of training samples, it might be a good idea to initialize your Embedding layer with word2vec weights, so at least your model recognizes that "Alps" and "Himalaya" are similar things, even if they don't both occur in sentences of your training dataset.

[1] How does Keras 'Embedding' layer work?

[2] https://www.tensorflow.org/tutorials/word2vec

[3] https://link.springer.com/article/10.1007/s10489-017-1109-7

NOTE: Actually, the image shows the activations of the layer after the Embedding layer, but for the purpose of this example it does not matter... See more details in [3]

$\endgroup$
  • 2
    $\begingroup$ This is exactly the explanation I was looking for! I think this sentence should be in bold: "Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words." $\endgroup$ – Kevin Jul 31 '18 at 15:05
  • $\begingroup$ thanks for clear explanation. I wanna use it in my paper so i should cite it to a paper. what are your references i.e which paper have you read and got this result? @Daniel López $\endgroup$ – maryam2013 Nov 28 '18 at 14:41
  • $\begingroup$ @maryam2013 please do not use answers for commenting. You don't heed anyone's approval for quoting his/her work. As about the content, it is all under CC-BY-SA license, so it can be re-used accordingly. $\endgroup$ – Tim Nov 28 '18 at 14:46
0
$\begingroup$

http://colah.github.io/posts/2014-07-NLP-RNNs-Representations -> this blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.

$\endgroup$
  • 3
    $\begingroup$ You should elaborate rather than mainly providing a link. $\endgroup$ – Michael R. Chernick Jan 22 at 7:07
0
$\begingroup$

The embedding layer is just a projection from discrete and sparse 1-hot-vector into a continuous and dense latent space. It is a matrix of (n,m) where n is your vocabulary size and n is your desired latent space dimensions. Only in practice, there's no need to actually do the matrix multiplication, and instead you can save on computation by using the index. So in practice, it is a layer that maps positive integers (indices corresponding to words) into dense vectors of fixed size (the embedding vectors).

You could train it to create a Word2Vec embedding by using Skip-Gram or CBOW. Or you can train it on your specific problem to get an embedding suited for your specific task at hand. You could also load pre-trained embeddings (like Word2Vec, GloVe etc.) and then continue training on your specific problem ( a form of transfer learning ).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.