What is an embedding layer in a neural network?

Question

In many neural network libraries, there are 'embedding layers', like in Keras or Lasagne.

I am not sure I understand its function, despite reading the documentation. For example, in the Keras documentation it says:

Turn positive integers (indexes) into denses vectors of fixed size, eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]

Could a knowledgeable person explain what it does, and when you would use it?

---

EDIT: Regarding pasting in documentation, there is not much to paste from the documentation, hence my question. I don't understand the transformation it does, nor why it should be used.

Anyway, this is how it's explained in Keras:

Embedding

keras.layers.embeddings.Embedding(input_dim, output_dim, init='uniform', input_length=None, weights=None, W_regularizer=None, W_constraint=None, mask_zero=False) Turn positive integers (indexes) into denses vectors of fixed size, eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]

Input shape: 2D tensor with shape: (nb_samples, sequence_length). Output shape: 3D tensor with shape: (nb_samples, sequence_length, output_dim). Arguments:

input_dim: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occurring in the input data. output_dim: int >= 0. Dimension of the dense embedding

And here it's how it's explained in Lasagne:

A layer for word embeddings. The input should be an integer type Tensor variable.

Parameters: incoming : a Layer instance or a tuple

The layer feeding into this layer, or the expected input shape.

input_size: int

The Number of different embeddings. The last embedding will have index input_size - 1.

output_size : int

The size of each embedding.

W : Theano shared variable, expression, numpy array or callable

Initial value, expression or initializer for the embedding matrix. This should be a matrix with shape (input_size, output_size). See lasagne.utils.create_param() for more information.

Examples

>>> from lasagne.layers import EmbeddingLayer, InputLayer, get_output
>>> import theano
>>> x = T.imatrix()
>>> l_in = InputLayer((3, ))
>>> W = np.arange(3*5).reshape((3, 5)).astype('float32')
>>> l1 = EmbeddingLayer(l_in, input_size=3, output_size=5, W=W)
>>> output = get_output(l1, x)
>>> f = theano.function([x], output)
>>> x_test = np.array([[0, 2], [1, 2]]).astype('int32')
>>> f(x_test) array([[[  0.,   1.,   2.,   3.,   4.],
                      [ 10.,  11.,  12.,  13.,  14.]],
                     [[  5.,   6.,   7.,   8.,   9.],
                      [ 10.,  11.,  12.,  13.,  14.]]], dtype=float32)

Answer 1

Relation to Word2Vec

==========================================

Word2Vec in a simple picture:

_{(source: netdna-ssl.com)}

More in-depth explanation:

I believe it's related to the recent Word2Vec innovation in natural language processing. Roughly, Word2Vec means our vocabulary is discrete and we will learn an map which will embed each word into a continuous vector space. Using this vector space representation will allow us to have a continuous, distributed representation of our vocabulary words. If for example our dataset consists of n-grams, we may now use our continuous word features to create a distributed representation of our n-grams. In the process of training a language model we will learn this word embedding map. The hope is that by using a continuous representation, our embedding will map similar words to similar regions. For example in the landmark paper Distributed Representations of Words and Phrases and their Compositionality, observe in Tables 6 and 7 that certain phrases have very good nearest neighbour phrases from a semantic point of view. Transforming into this continuous space allows us to use continuous metric notions of similarity to evaluate the semantic quality of our embedding.

Explanation using Lasagne code

Let's break down the Lasagne code snippet:

x = T.imatrix()

x is a matrix of integers. Okay, no problem. Each word in the vocabulary can be represented an integer, or a 1-hot sparse encoding. So if x is 2x2, we have two datapoints, each being a 2-gram.

l_in = InputLayer((3, ))

The input layer. The 3 represents the size of our vocabulary. So we have words $w_0, w_1, w_2$ for example.

W = np.arange(3*5).reshape((3, 5)).astype('float32')

This is our word embedding matrix. It is a 3 row by 5 column matrix with entries 0 to 14.

Up until now we have the following interpretation. Our vocabulary has 3 words and we will embed our words into a 5 dimensional vector space. For example, we may represent one word $w_0 = (1,0,0)$, and another word $w_1 = (0, 1, 0)$ and the other word $w_2 = (0, 0, 1)$, e.g. as hot sparse encodings. We can view the $W$ matrix as embedding these words via matrix multiplication. Therefore the first word $w_0 \rightarrow w_0W = [0, 1, 2, 3, 4].$ Simmilarly $w_1 \rightarrow w_1W = [5, 6, 7, 8, 9]$.

It should be noted, due to the one-hot sparse encoding we are using, you also see this referred to as table lookups.

l1 = EmbeddingLayer(l_in, input_size=3, output_size=5, W=W)

The embedding layer

 output = get_output(l1, x)

Symbolic Theano expression for the embedding.

f = theano.function([x], output)

Theano function which computes the embedding.

x_test = np.array([[0, 2], [1, 2]]).astype('int32')

It's worth pausing here to discuss what exactly x_test means. First notice that all of x_test entries are in {0, 1, 2}, i.e. range(3). x_test has 2 datapoints. The first datapoint [0, 2] represents the 2-gram $(w_0, w_2)$ and the second datapoint represents the 2-gram $(w_1, w_2)$.

We wish to embed our 2-grams using our word embedding layer now. Before we do that, let's make sure we're clear about what should be returned by our embedding function f. The 2 gram $(w_0, w_2)$ is equivalent to a [[1, 0, 0], [0, 0, 1]] matrix. Applying our embedding matrix W to this sparse matrix should yield: [[0, 1, 2, 3, 4], [10, 11, 12, 13, 14]]. Note in order to have the matrix multiplication work out, we have to apply the word embedding matrix $W$ via right multiplication to the sparse matrix representation of our 2-gram.

f(x_test) 

returns:

          array([[[  0.,   1.,   2.,   3.,   4.],
                  [ 10.,  11.,  12.,  13.,  14.]],
                 [[  5.,   6.,   7.,   8.,   9.],
                  [ 10.,  11.,  12.,  13.,  14.]]], dtype=float32)

To convince you that the 3 does indeed represent the vocabulary size, try inputting a matrix x_test = [[5, 0], [1, 2]]. You will see that it raises a matrix mis-match error.

Answer 2

In https://stackoverflow.com/questions/45649520/explain-with-example-how-embedding-layers-in-keras-works/ I tried to prepare an example using 2 sentences, keras's texts_to_sequences

'This is a text' --> [0 0 1 2 3 4]

and embedding layer. Based on How does Keras 'Embedding' layer work? the embedding layer first initialize the embedding vector at random and then uses network optimizer to update it similarly like it would do to any other network layer in keras.

[0 0 1 2 3 4] --> 
[-0.01494285, -0.007915  ,  0.01764857],
[-0.01494285, -0.007915  ,  0.01764857],
[-0.03019481, -0.02910612,  0.03518577],
[-0.0046863 ,  0.04763055, -0.02629668],
[ 0.02297204,  0.02146662,  0.03114786],
[ 0.01634104,  0.02296363, -0.02348827]

Above would be some initial embeding vector for a sentence of (maximum) 6 words and output_dim of 3.