27
$\begingroup$

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

$\endgroup$
1

3 Answers 3

37
$\begingroup$

Embedding layers in Keras are trained just like any other layer in your network architecture: they are tuned to minimize the loss function by using the selected optimization method. The major difference with other layers, is that their output is not a mathematical function of the input. Instead the input to the layer is used to index a table with the embedding vectors [1]. However, the underlying automatic differentiation engine has no problem to optimize these vectors to minimize the loss function...

So, you cannot say that the Embedding layer in Keras is doing the same as word2vec [2]. Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words. With Keras's embedding layer, you are just trying to minimize the loss function, so if for instance you are working with a sentiment classification problem, the learned embedding will probably not capture complete word semantics but just their emotional polarity...

For example, the following image taken from [3] shows the embedding of three sentences with a Keras Embedding layer trained from scratch as part of a supervised network designed to detect clickbait headlines (left) and pre-trained word2vec embeddings (right). As you can see, word2vec embeddings reflect the semantic similarity between phrases b) and c). Conversely, the embeddings generated by Keras's Embedding layer might be useful for classification, but do not capture the semantical similarity of b) and c).

enter image description here

This explains why when you have a limited amount of training samples, it might be a good idea to initialize your Embedding layer with word2vec weights, so at least your model recognizes that "Alps" and "Himalaya" are similar things, even if they don't both occur in sentences of your training dataset.

[1] How does Keras 'Embedding' layer work?

[2] https://www.tensorflow.org/tutorials/word2vec

[3] https://link.springer.com/article/10.1007/s10489-017-1109-7

NOTE: Actually, the image shows the activations of the layer after the Embedding layer, but for the purpose of this example it does not matter... See more details in [3]

$\endgroup$
2
  • 3
    $\begingroup$ This is exactly the explanation I was looking for! I think this sentence should be in bold: "Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words." $\endgroup$
    – Kevin
    Jul 31, 2018 at 15:05
  • $\begingroup$ @nice answer, you answer here and answer here helped me a lot! $\endgroup$
    – Haitao Du
    Apr 1, 2022 at 17:03
7
$\begingroup$

The embedding layer is just a projection from discrete and sparse 1-hot-vector into a continuous and dense latent space. It is a matrix of size (n,m) where n is your vocabulary size and m is your desired latent space dimensions. Only in practice, there's no need to actually do the matrix multiplication, and instead you can save on computation by using the index. So in practice, it is a layer that maps positive integers (indices corresponding to words) into dense vectors of fixed size (the embedding vectors).

You could train it to create a Word2Vec embedding by using Skip-Gram or CBOW. Or you can train it on your specific problem to get an embedding suited for your specific task at hand. You could also load pre-trained embeddings (like Word2Vec, GloVe etc.) and then continue training on your specific problem (a form of transfer learning).

$\endgroup$
0
$\begingroup$

one way to envision the implementation (not sure this is how Keras/Pytorch or other frameworks implement it), such that the weight could be "learned" by back propagation, is to map the Embedding Matrix (n_vocab x n_emb_dim) as the weight between the input layer where word/token are one-hot encoded, and the embedded-layer where each token is represented as a n_emb_dim vector.

Given that the One-hot layer dimension is n_vocab, and the embedded layer dimension is n_emb_dim, the number of weights for the full connections between these two layers is n_vocab x n_emb_dim, which is the same as the "lookup table of weights".

This is only for one token, what if our input is a sequence of token, like a sentence? We could easily duplicate and concatenate the embedding layer vertically, with weight-sharing, so the weights count remain the same regardless of the sequence length.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.