I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-LSTM language model then use the internal representations from this model to produce "ELMo" representations. These representations are given by the formula:

$$\text{ELMo}_k^{task} = \gamma^{task}~\sum_{j=0}^L s_j^{task}\textbf{h}_{k,j}^{LM}$$


  • $\quad \gamma^{task}$ and $s_j^{task}$ are task specific parameters
  • $\quad \textbf{h}_{k,0}^{LM}$ is the token representation of token $k$
  • $\quad \textbf{h}_{k,j>0}^{LM}$ is the biLSTM representation of token $k$

In the implementation available on Tensorflow Hub we can see that the token representation has a size of 512 while the biLSTM representations have a size 1024.

enter image description here

I don't understand how they are making the sum over vectors that have different sizes. Am I missing something here?


1 Answer 1


The word embeddings are inputs to both the forward LM and the backward LM. This is stated explicitly in the paper, but I like the visualization of it in Figure 3 of Devlin et al. (2019). For the sake of averaging, they nevertheless treat the forward and backward representations as distinct, so word_emb shows up twice.

The lstm_outputs_1 and lstm_outputs_2 values are concatenations of the forward and backward hidden states, for each layer (1 and 2). Each LM has a 512-dimensional hidden state. When the authors perform averaging, they essentially concatenate the 512-dimensional word embeddings word_emb to themselves to make a 1024-dimensional vector per word. (Think of it however you want: either (a) this is the base layer, so they have three 1024-dimensional representations that they average, or (b) they have a forward model whose 3 representations they average, then the same for the backward model, and they concatenate the averaged forward and averaged backward representations.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.