I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, t_{2}, ..., t_{N})$ we seek to compute the probability of the sequence by estimating the probability of observing each token given the sequence, or $p(t_{k}|(t_{1}, t_{2}, ..., t_{N}))$.
Specifically, we seek to maximize the joint log-likelihood function of the forward and backward directions: $$ \sum^{N}_{k=1}(\space \log \space p(t_{k} | t_{1},...,t_{k−1};Θ_{x}, Θ_{LSTM}^{\rightarrow},Θ_{s}) + \log \space p(t_{k} | t_{k+1},...,t_{N};Θ_{x}, Θ_{LSTM}^{\leftarrow},Θ_{s})\space) $$
Where $Θ_{x}$ computes a context-independent representation $x_{k}$ for each $t_{k}$, passes it to intermediate layers $Θ_{LSTM}^{\rightarrow}$ and $Θ_{LSTM}^{\leftarrow}$, which then pass their output to softmax layer $Θ_{s}$.
My question is: how do we find context independent token representations $x$ in layer $Θ_{x}$? I'm unclear on that point.