5
$\begingroup$

Recently, I have been looking at seq2seq models that have been used for translating from one language to another using recurrent neural networks (often with LSTM cells).

Those models can also be used to generate text, one character at a time. Based on its internal memory, which efficiently encodes the previous characters, the model learns a probability distribution for the next character.

When looking at the various implementations of these seq2seq models, like this one, I see an embedding matrix is trained jointly with the neural network. As I understand it, each of this matrix's rows is the 'embedding' of a particular character (each character is represented by an integer: its id in a finite vocabulary).

What is the rationale behind using this embedding? What is it used for? Why is it needed?

LSTM: Long-Short Term Memory

Improve this question
$\endgroup$

1 Answer 1

Reset to default
7
$\begingroup$

Embeddings are dense vector representations of the characters. The rationale behind using it is to convert an arbitrary discrete id, to a continuous representation.

The main advantage is that back-propagation is possible over continuous representations while it is not over discrete representations. A second advantage is that the vector representation might contain additional information based on its location compared to the other characters.

This is still a hot area of research. If you are interested in learning more, check out the word2vec algorithms: vector embeddings are learned for words where interesting relationships are learned. For example an interesting write-up here: https://deeplearning4j.org/word2vec.html

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your very clear answer. As I understand it, the embedding matrix is initialized randomly and then modified during training, making it become something meaningful (word2vec is a good example of that). One thing I don't understand however is up to where does the error back-propagate? The actual input of the model is a sequence of integers (character ids). Does the error back-propagate up to those character ids? Or does it back-propagate only up to the embedding of those character ids? $\endgroup$
    – Valentin Iovene
    Commented Mar 22, 2017 at 16:31
  • $\begingroup$ @GR4, do you have any comment on how to choose the embedding size, esp. for characters given the character vocabulary is usually much much smaller than a word vocabulary, or recommend any papers? $\endgroup$
    – zyxue
    Commented May 25, 2018 at 3:27
  • 1
    $\begingroup$ @ValentinIovene errors are back-propagated up to the input (character ids) $\endgroup$
    – GR4
    Commented May 27, 2018 at 13:50
  • $\begingroup$ @zyxue typical embedding sizes for words are around 100 - 400 (you can check some of mikolov's papers on embedding techniques if you want to go more in detail). Not sure about embedding sizes for characters but in general you could choose the embedding size based on performance of the validation set. $\endgroup$
    – GR4
    Commented May 27, 2018 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.