Need to understand the working of 'Embedding' layer in Keras library. I execute the following code in Python

import numpy as np
from keras.models import Sequential
from keras.layers import Embedding

model = Sequential()
model.add(Embedding(5, 2, input_length=5))

input_array = np.random.randint(5, size=(1, 5))

model.compile('rmsprop', 'mse')
output_array = model.predict(input_array)

which gives the following output

input_array = [[4 1 3 3 3]]
output_array = 
[[[ 0.03126476  0.00527241]
  [-0.02369716 -0.02856163]
  [ 0.0055749   0.01492429]
  [ 0.0055749   0.01492429]
  [ 0.0055749   0.01492429]]]

I understand that each value in the input_array is mapped to 2 element vector in the output_array, so a 1 X 4 vector gives 1 X 4 X 2 vectors. But how are the mapped values computed?


3 Answers 3


In fact, the output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all possible vectors. That is the reason why you need to specify the size of the vocabulary as the first argument (so the table can be initialized).

The most common application of this layer is for text processing. Let's see a simple example. Our training set consists only of two phrases:

Hope to see you soon

Nice to see you again

So we can encode these phrases by assigning each word a unique integer number (by order of appearance in our training dataset for example). Then our phrases could be rewritten as:

[0, 1, 2, 3, 4]

[5, 1, 2, 3, 6]

Now imagine we want to train a network whose first layer is an embedding layer. In this case, we should initialize it as follows:

Embedding(7, 2, input_length=5)

The first argument (7) is the number of distinct words in the training set. The second argument (2) indicates the size of the embedding vectors. The input_length argument, of course, determines the size of each input sequence.

Once the network has been trained, we can get the weights of the embedding layer, which in this case will be of size (7, 2) and can be thought as the table used to map integers to embedding vectors:

|   index    |  Embedding |
|     0      | [1.2, 3.1] |
|     1      | [0.1, 4.2] |
|     2      | [1.0, 3.1] |
|     3      | [0.3, 2.1] |
|     4      | [2.2, 1.4] |
|     5      | [0.7, 1.7] |
|     6      | [4.1, 2.0] |

So according to these embeddings, our second training phrase will be represented as:

[[0.7, 1.7], [0.1, 4.2], [1.0, 3.1], [0.3, 2.1], [4.1, 2.0]]

It might seem counterintuitive at first, but the underlying automatic differentiation engines (e.g., Tensorflow or Theano) manage to optimize these vectors associated with each input integer just like any other parameter of your model.

For an intuition of how this table lookup is implemented as a mathematical operation which can be handled by the automatic differentiation engines, consider the embeddings table from the example as a (7, 2) matrix. Then, for a given word, you create a one-hot vector based on its index and multiply it by the embeddings matrix, effectively replicating a lookup. For instance, for the word "soon" the index is 4, and the one-hot vector is [0, 0, 0, 0, 1, 0, 0]. If you multiply this (1, 7) matrix by the (7, 2) embeddings matrix you get the desired two-dimensional embedding, which in this case is [2.2, 1.4].

It is also interesting to use the embeddings learned by other methods/people in different domains (see https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html) as done in [1].

[1] López-Sánchez, D., Herrero, J. R., Arrieta, A. G., & Corchado, J. M. Hybridizing metric learning and case-based reasoning for adaptable clickbait detection. Applied Intelligence, 1-16.

  • 13
    $\begingroup$ Thank you for the answer. Just one query that how are the weights of the embedding layer obtained. Like for the index 0, how is [1.2, 3.1] obtained. $\endgroup$
    – prashanth
    Commented Sep 29, 2017 at 15:39
  • 18
    $\begingroup$ The contents of the table that relates indexes to embedding vectors (i.e., the weights of the embedding layer) are initialized at random and then optimized by the training algorithm (e.g., Gradient Descent). $\endgroup$ Commented Sep 29, 2017 at 19:27
  • 10
    $\begingroup$ Thanks. I'm still a bit unclear what the optimizer would optimizer against? Like, what is the "correct answer" that allows it to compute a loss function? Or said another way, what is it doing for the forward and backward pass? $\endgroup$
    – bwest87
    Commented Dec 5, 2017 at 0:28
  • 9
    $\begingroup$ so ... embedding is basically just a subnetwork of the overall architecture which reduces any one-hot encoded inputs down into fewer inputs, afaict.. $\endgroup$ Commented Dec 14, 2017 at 9:55
  • 3
    $\begingroup$ @mikalai, To the best of my knowledge, only the embeddings corresponding to words that appear in the training set get updated during the training process. If you initialize the embedding layer with additional embeddings for words that are not in the training set these will remain as initialized during the training process. Also, I have noticed that Keras implementation raises an exception when you ask the embedding layer to process a sequence with an integer which is not in the table (i.e., a new word). This is something important to keep in mind when pre-processing. $\endgroup$ Commented Oct 17, 2018 at 9:35

I also had the same question and after reading a couple of posts and materials I think I figured out what embedding layer role is.

I think this post is also helpful to understand, however, I really find Daniel's answer convenient to digest. But I also got the idea behind it mainly by understanding the embedding words.

I believe it's inaccurate to say embedding layers reduce one-hot encoding input down to fewer inputs. After all the one-hot vector is a one-dimensional data and it is indeed turned into 2 dimensions in our case. Better to be said that

embedding layer comes up with a relation of the inputs in another dimension

Whether it's in 2 dimensions or even higher.

I also find a very interesting similarity between word embedding to the Principal Component Analysis. Although the name might look complicated the concept is straightforward. What PCA does is to define a set of data based on some general rules (so-called principle components). So it's like having a data and you want to describe it but using only 2 components. Which in this sense is very similar to word embeddings. They both do the same-alike job in different context. You can find out more here. I hope maybe understanding PCA helps understanding embedding layers through analogy.

To wrap up, the answer to the original question of the post that "how does it calculate the value?" would be:

  • Basically, our neural network captures underlying structure of the inputs (our sentences) and puts relation between words in our vocabulary into a higher dimension (let's say 2) by optimization.
  • Deeper understanding would say that the frequency of each word appearing with another word from our vocabulary influences (in a very naive approach we can calculate it by hand)
  • Aforementioned frequency could be one of many underlying structures that NN can capture
  • You can find the intuition on the youtube link explaining the word embeddings
  • 10
    $\begingroup$ Nice point of view. However, I think it is worth noting that while word-embedding techniques such as word2vec try to capture the full meaning of words in the resulting embedding, the embedding layer in a supervised network might not learn such a semantically-rich and general representation. For example, if your network is trained to do sentiment classification, it will probably just group/cluster words in the embedding according to their "emotional" load. Nevertheless, based on my experience it is often useful to initialize your embedding layer with weights learned by word2vec on a big corpus. $\endgroup$ Commented Jan 4, 2018 at 10:08
  • 6
    $\begingroup$ one-hot vector is not one dimensional data. Its dimension is size of the vocabulary. $\endgroup$
    – Binu Jasim
    Commented Jan 7, 2018 at 22:42
  • 3
    $\begingroup$ @BinuJasim you're right. The one-hot vectors representing a vocabulary is not a one-dimensional data. But the information that it represents is indeed a one dimensional and every entity within the vocabulary is a one-dimensional data. It's true that we have n*w (n = vocabulary size, w = number of bits) elements but each binary value represents a vector which again is a one-dimensional input. $\endgroup$ Commented Jan 15, 2018 at 14:53

If you're more interested in the "mechanics", the embedding layer is basically a matrix which can be considered a transformation from your discrete and sparse 1-hot-vector into a continuous and dense latent space. Only to save the computation, you don't actually do the matrix multiplication, as it is redundant in the case of 1-hot-vectors.

So, say you have a vocabulary size of 5000, as your input dimension - and you want to find a 256 dimension output representation of it - you will have a (5000,256) shape matrix, which you "should" multiply your 1-hot vector representation to get the latent vector. Only in practice instead of multiplying you just take the index...

Source: Andrew Ng

(One way that helps me think of it in theory, is as a Dense layer only without bias or activation... )

The weights of this matrix are learned through training - you could train it as a Word2Vec, GloVe, etc. - or on the specific task that you are dealing with. Or you can load pre-trained weights (say GloVe) and continue training on your specific task.


Not the answer you're looking for? Browse other questions tagged or ask your own question.