I'll explain this in the case of a standard LSTM language model. The task is to predict the next symbol (either a word or a character) given the previous symbols. When the input is The quick brown, we want the network to predict fox for a word-level model. When the input is T h e <space> q u i c k <space> b r o w n <space>, we want the network to predict f.
The outline of this network is
- Embedding (input has dimension $p_0$, output has dimension $p_1$)
- Linear layer (output has dimension $p_2$)
- LSTM layers (output has dimension $p_2$)
- Linear layer (output has dimension $p_0$)
- Softmax (output has dimension $p_0$)
I'll explain each part in more detail.
So for the input The quick brown fox jumped over the lazy dog, we embed each of the words using either a pre-trained embedding or an embedding that we train ourselves. This gives vectors $v_\text{the}, v_\text{quick}, v_\text{brown}, \dots v_\text{dog}$. Each of the $v_i$ has dimension $p_1$.
The core of the LSTM language model is 1 or more LSTM layers with dimension $p_2$. In general, $p_1\neq p_2$, so whenever this is the case, we can put a linear layer between the embedding and LSTM layers so that the input is the appropriate size.
The LSTM prediction is based on three inputs: $x(t), c(t-1), h(t-1)$. The $c$ and $h$ are created recursively by the LSTM. The $x(t)$ contains the embedding, optionally transformed by the linear layer.
In the language model, the $c$ values are only used by the LSTM to predict the next step in the sequence. The $h$ values are the LSTM's "predictions" for the next token in the sequence. Because the $h$ values might not have the same dimension as the vocabulary, there's usually a linear layer to project $h$ with dimension $p_2$ to the vocabulary dimension $p_3$.
How would the Encoder (if the cell is part of the encoder) train the weights applied to the inputs inside them, when it does not make predictions or in general computes loss?
The LSTM does make "predictions," in a loose sense -- it produces vectors with size $p_2$, which might need to be projected to size $p_0$.
The predictions are $h$. The weights are updated using backpropagation.
And if it would be part of the Decoder: Where would it get the xs. Or are the xs simply the same that are put into the Encoder?
The $x$ values are the inputs to the layer, so the $x$ values come from you, the person who programmed the network. In this example, the $x$ values come from the embedding.
According to my information, these cells exist in the Encoder and in the Decoder. But:
