When should an embedding layer be used? How big should an embedding be?

Question

I am currently in the process of learning about seq2seq autoencoders for a task involving sentence embedding (samples are sentences, with words represented as integers in a vocab of size $n$). In the different tutorials and examples I have viewed I have noticed some authors use an embedding layer after the input layer while others do not. Now I understand what embedding layers are and roughly how they work, but I can not seem to figure out when one should use an embedding layer - specifically in the context of my task.

My best guess is that embedding layers simply make the representation of the data easier for the network to work with, transforming a large vocab of $n$ words as integers into fixed sized float vectors. However I do not really have any evidence to back this up other than intuition.

So are there any rules of thumb as to when embedding layer after a network's input layer? If so, are there any good rules of thumb for what embedding size to choose in proportion to the vocabulary size?

If it matters, I am referring to the Embedding layer in Tensorflow 2 / Keras.

Answer 1

Embedding layers and multiplication by appropriate 1-hot vectors can be shown to be the same in the sense that they produce the same result. See: How does torch.nn.Embedding or tf.keras.layers.Embedding compare to a dense layer?

Computing the dot product of a $1$-hot vector with a large matrix is expensive, and not strictly necessary. Computing a bunch of these (because you have several words in a sentence) is wasteful because multiplying by zero a bunch of times is silly. Instead we can just select the hot entries from the matrix directly, which is what an embedding layer does.

If you're not convinced, write down a matrix and a 1-hot vector and carry out the multiplication.

---

are there any rules of thumb as to when embedding layer after a network's input layer?

Since the two methods yield the same result, we could distinguish them on the basis of which is faster. Which method wins the race is a specific to the software, which is not on-topic here.

---

are there any good rules of thumb for what embedding size to choose in proportion to the vocabulary size?

This Google Developers Blog post "Introducing TensorFlow Feature Columns" suggests using the fourth root of the number of categories (vocabulary size).

$$ \text{embedding dimensions} = \text{number of categories}^{\frac{1}{4}} $$

I'm not aware of any results establish that this rule of thumb is optimal. Different tasks or applications might require different embeddings. Of course, engineering constraints, such as storage or computational requirements, might limit the size of the embedding dimension.