# How are the internal representations in ELMo averaged?

I have been reading the paper "Deep contextualized word representations" (by Peters et al, 2018) to learn about the new embedding method called ELMo. In this paper, the authors train a charCNN + bi-LSTM language model then use the internal representations from this model to produce "ELMo" representations. These representations are given by the formula:

$$\text{ELMo}_k^{task} = \gamma^{task}~\sum_{j=0}^L s_j^{task}\textbf{h}_{k,j}^{LM}$$

where:

• $\quad \gamma^{task}$ and $s_j^{task}$ are task specific parameters
• $\quad \textbf{h}_{k,0}^{LM}$ is the token representation of token $k$
• $\quad \textbf{h}_{k,j>0}^{LM}$ is the biLSTM representation of token $k$

In the implementation available on Tensorflow Hub we can see that the token representation has a size of 512 while the biLSTM representations have a size 1024.

I don't understand how they are making the sum over vectors that have different sizes. Am I missing something here?

• Could be that word_emb is just broadcasted since the LSTMs are bidirectional, but not sure. May be worth looking at the actual code: github.com/allenai/bilm-tf/blob/… Commented Jan 24, 2020 at 14:56

The word embeddings are inputs to both the forward LM and the backward LM. This is stated explicitly in the paper, but I like the visualization of it in Figure 3 of Devlin et al. (2019). For the sake of averaging, they nevertheless treat the forward and backward representations as distinct, so word_emb shows up twice.
The lstm_outputs_1 and lstm_outputs_2 values are concatenations of the forward and backward hidden states, for each layer (1 and 2). Each LM has a 512-dimensional hidden state. When the authors perform averaging, they essentially concatenate the 512-dimensional word embeddings word_emb to themselves to make a 1024-dimensional vector per word. (Think of it however you want: either (a) this is the base layer, so they have three 1024-dimensional representations that they average, or (b) they have a forward model whose 3 representations they average, then the same for the backward model, and they concatenate the averaged forward and averaged backward representations.)