I understand word embeddings and word2vec.

In this paper: https://arxiv.org/pdf/1603.01547.pdf

they are saying a new type of word embedding.

Our model uses one word embedding function
and two encoder functions. The word embedding
function e translates words into vector representations.
The first encoder function is a document
encoder f that encodes *every word from the document*
d *in the context of the whole document*.
We call this the **contextual embedding**.

Is this some new way of encoding, How can I implement this? Thanks .

  • $\begingroup$ Where do they claim it's a new type of word embedding? $\endgroup$ Oct 11, 2016 at 14:37

1 Answer 1


The contextual embedding of a word is just the corresponding hidden state of a bi-GRU:

In our model the document encoder $f$ is implemented as a bidirectional Gated Recurrent Unit (GRU) network whose hidden states form the contextual word embeddings, that is $f_i(d) = \overrightarrow{f_i}(d) \,\, ||\,\, \overleftarrow{f_i}(d)$, where $||$ denotes vector concatenation and $\overrightarrow{f_i}$ and $\overleftarrow{f_i}$ denote forward and backward contextual embeddings from the respective recurrent networks.

In red is the contextual embedding of the first word:

enter image description here

  • $\begingroup$ Thanks a lot for the answer. I do not understand why is it called contextutal embedding? Does it really capture the context? $\endgroup$
    – Sie Tw
    Oct 12, 2016 at 15:52
  • $\begingroup$ @SieTw Yes, it captures the context, since the hidden states are computed based on the previous hidden states. $\endgroup$ Oct 12, 2016 at 18:02
  • $\begingroup$ source of the diagram? $\endgroup$
    – aerin
    Dec 26, 2018 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.