I've been reading this paper on ELMo word representations.

For context, here's my understanding of the standard bi-directional language model (biLM) thus far:

Given a sequence of tokens $(t_{1}, t_{2}, ..., t_{N})$ we seek to compute the probability of the sequence by estimating the probability of observing each token given the sequence, or $p(t_{k}|(t_{1}, t_{2}, ..., t_{N}))$.

Specifically, we seek to maximize the joint log-likelihood function of the forward and backward directions: $$ \sum^{N}_{k=1}(\space \log \space p(t_{k} | t_{1},...,t_{k−1};Θ_{x}, Θ_{LSTM}^{\rightarrow},Θ_{s}) + \log \space p(t_{k} | t_{k+1},...,t_{N};Θ_{x}, Θ_{LSTM}^{\leftarrow},Θ_{s})\space) $$
Where $Θ_{x}$ computes a context-independent representation $x_{k}$ for each $t_{k}$, passes it to intermediate layers $Θ_{LSTM}^{\rightarrow}$ and $Θ_{LSTM}^{\leftarrow}$, which then pass their output to softmax layer $Θ_{s}$.

My question is: how do we find context independent token representations $x$ in layer $Θ_{x}$? I'm unclear on that point.


1 Answer 1


I also have the similar question when reading the "deep contextual" paper. After checking the reference listed in the paper, I think the context independent token representations is trained based on character level RNN that for represent each token with a feature vector.

I found a concise but clean version of tutorial talking about the paper: https://medium.com/@ayush2503/contextualized-word-representations-5df54663323f

it also mentions that

Token Representation: A context-independent token representation is computed with character convolutions.

Best, Eigen


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.