Embedding layers are a common choice to map some high-dimensional, discrete input to real-valued (computationally represented using floating point) numbers in a much smaller number of dimensions. Some common usages are word embeddings, character embeddings, byte embeddings, categorical embeddings, or entity embeddings.

Common implementations such as torch.nn.Embedding and tf.keras.layers.Embedding accept an integer as an input and yield a floating-point vector as an output.

Is an embedding layer comparable to a dense layer? How?


1 Answer 1


Using an embedding layer is equivalent to using a dense layer with a very specific type of input.

First, some notation. You have $n$ symbols (characters, words, bytes, or some other categorical data) that you wish to embed (represent) as a floating point vector with $m$ elements. We can collect all of these floating point vectors in a matrix $A$ of shape $m \times n$. A vector $x$ has $n$ elements in total, where $n-1$ elements are 0, and 1 element at position $i$ has the value 1. This is sometimes called a "one-hot vector," because it has only one nonzero element.

By construction, the matrix-vector product $Ax$ will select (copy) the $i$th row from the matrix $A$, because only the $i$th row has nonzero values, and that value is 1. (If you're unsure of this, write down some small matrix $A$ and a one-hot vector $x$ and carry out the arithmetic.)

These facts describe the equivalence between a dense layer (with a special kind of input) and an embedding layer. However, the software implementations work slightly differently, because they take an integer as an input. Instead of a one-hot vector with 1 in element $i$, the embedding layers take the index $i$ as the input, then just selects that row from $A$ and returns it. This is why embedding layers are often described as "lookup tables": they retrieve values according to the input, the index of the array.

These kinds of embedding layers are more computationally efficient because it is cheaper to simply select the desired values. By contrast, naïvely carrying out the matrix-vector product $Ax$ is wasteful; there are $nm$ multiplications, and almost all of them are multiplication by 0. Additionally, storing the single integer is more memory efficient than storing the whole one-hot vector.

  • 2
    $\begingroup$ +1. Also, using the index instead of the one-hot vector provides space savings. $\endgroup$ Commented Oct 2, 2021 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.