In many neural network libraries, there are 'embedding layers', like in Keras or Lasagne.
I am not sure I understand its function, despite reading the documentation. For example, in the Keras documentation it says:
Turn positive integers (indexes) into denses vectors of fixed size, eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
Could a knowledgeable person explain what it does, and when you would use it?
EDIT: Regarding pasting in documentation, there is not much to paste from the documentation, hence my question. I don't understand the transformation it does, nor why it should be used.
Anyway, this is how it's explained in Keras:
Embedding
keras.layers.embeddings.Embedding(input_dim, output_dim, init='uniform', input_length=None, weights=None, W_regularizer=None, W_constraint=None, mask_zero=False) Turn positive integers (indexes) into denses vectors of fixed size, eg. [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]
Input shape: 2D tensor with shape: (nb_samples, sequence_length). Output shape: 3D tensor with shape: (nb_samples, sequence_length, output_dim). Arguments:
input_dim: int >= 0. Size of the vocabulary, ie. 1+maximum integer index occurring in the input data. output_dim: int >= 0. Dimension of the dense embedding
And here it's how it's explained in Lasagne:
A layer for word embeddings. The input should be an integer type Tensor variable.
Parameters: incoming : a Layer instance or a tuple
The layer feeding into this layer, or the expected input shape.
input_size: int
The Number of different embeddings. The last embedding will have index input_size - 1.
output_size : int
The size of each embedding.
W : Theano shared variable, expression, numpy array or callable
Initial value, expression or initializer for the embedding matrix. This should be a matrix with shape (input_size, output_size). See lasagne.utils.create_param() for more information.
Examples
>>> from lasagne.layers import EmbeddingLayer, InputLayer, get_output >>> import theano >>> x = T.imatrix() >>> l_in = InputLayer((3, )) >>> W = np.arange(3*5).reshape((3, 5)).astype('float32') >>> l1 = EmbeddingLayer(l_in, input_size=3, output_size=5, W=W) >>> output = get_output(l1, x) >>> f = theano.function([x], output) >>> x_test = np.array([[0, 2], [1, 2]]).astype('int32') >>> f(x_test) array([[[ 0., 1., 2., 3., 4.], [ 10., 11., 12., 13., 14.]], [[ 5., 6., 7., 8., 9.], [ 10., 11., 12., 13., 14.]]], dtype=float32)