# How the embedding layer is trained in Keras Embedding layer

How is the embedding layer trained in Keras Embedding layer? (say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

Embedding layers in Keras are trained just like any other layer in your network architecture: they are tuned to minimize the loss function by using the selected optimization method. The major difference with other layers, is that their output is not a mathematical function of the input. Instead the input to the layer is used to index a table with the embedding vectors [1]. However, the underlying automatic differentiation engine has no problem to optimize these vectors to minimize the loss function...

So, you cannot say that the Embedding layer in Keras is doing the same as word2vec [2]. Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words. With Keras's embedding layer, you are just trying to minimize the loss function, so if for instance you are working with a sentiment classification problem, the learned embedding will probably not capture complete word semantics but just their emotional polarity...

For example, the following image taken from [3] shows the embedding of three sentences with a Keras Embedding layer trained from scratch as part of a supervised network designed to detect clickbait headlines (left) and pre-trained word2vec embeddings (right). As you can see, word2vec embeddings reflect the semantic similarity between phrases b) and c). Conversely, the embeddings generated by Keras's Embedding layer might be useful for classification, but do not capture the semantical similarity of b) and c).

This explains why when you have a limited amount of training samples, it might be a good idea to initialize your Embedding layer with word2vec weights, so at least your model recognizes that "Alps" and "Himalaya" are similar things, even if they don't both occur in sentences of your training dataset.

NOTE: Actually, the image shows the activations of the layer after the Embedding layer, but for the purpose of this example it does not matter... See more details in [3]

• This is exactly the explanation I was looking for! I think this sentence should be in bold: "Remember that word2vec refers to a very specific network setup which tries to learn an embedding which captures the semantics of words." – Kevin Jul 31 '18 at 15:05

The embedding layer is just a projection from discrete and sparse 1-hot-vector into a continuous and dense latent space. It is a matrix of (n,m) where n is your vocabulary size and n is your desired latent space dimensions. Only in practice, there's no need to actually do the matrix multiplication, and instead you can save on computation by using the index. So in practice, it is a layer that maps positive integers (indices corresponding to words) into dense vectors of fixed size (the embedding vectors).

You could train it to create a Word2Vec embedding by using Skip-Gram or CBOW. Or you can train it on your specific problem to get an embedding suited for your specific task at hand. You could also load pre-trained embeddings (like Word2Vec, GloVe etc.) and then continue training on your specific problem ( a form of transfer learning ).