How to compute context-independent token representations in a biLM?

Question

I've been reading this paper on ELMo word representations.

For context, here's my understanding of the standard bi-directional language model (biLM) thus far:

Given a sequence of tokens $(t_{1}, t_{2}, ..., t_{N})$ we seek to compute the probability of the sequence by estimating the probability of observing each token given the sequence, or $p(t_{k}|(t_{1}, t_{2}, ..., t_{N}))$.

Specifically, we seek to maximize the joint log-likelihood function of the forward and backward directions: $$ \sum^{N}_{k=1}(\space \log \space p(t_{k} | t_{1},...,t_{k−1};Θ_{x}, Θ_{LSTM}^{\rightarrow},Θ_{s}) + \log \space p(t_{k} | t_{k+1},...,t_{N};Θ_{x}, Θ_{LSTM}^{\leftarrow},Θ_{s})\space) $$
Where $Θ_{x}$ computes a context-independent representation $x_{k}$ for each $t_{k}$, passes it to intermediate layers $Θ_{LSTM}^{\rightarrow}$ and $Θ_{LSTM}^{\leftarrow}$, which then pass their output to softmax layer $Θ_{s}$.

My question is: how do we find context independent token representations $x$ in layer $Θ_{x}$? I'm unclear on that point.

Answer 1

I also have the similar question when reading the "deep contextual" paper. After checking the reference listed in the paper, I think the context independent token representations is trained based on character level RNN that for represent each token with a feature vector.

I found a concise but clean version of tutorial talking about the paper: https://medium.com/@ayush2503/contextualized-word-representations-5df54663323f

it also mentions that

Token Representation: A context-independent token representation is computed with character convolutions.

Best, Eigen