I'm trying to re-implement Transformer on my own but I'm struggling on the input encoding.
It seems that the original paper(Attention is All You Need) mentioned that they use Byte-pair Encoding to encode words into vectors but as I understand BPE is just used for word segmentation (compress size of the dictionary) so how will it produce a numeric vector?
In many implementations, I saw that they just initialled random weights for word embedding. In this case, will the weights be updated when the model is being trained?
Lastly, As they mentioned that "we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation", I understand that two embedding layers could be shared but why "the pre-softmax linear transformation" should be the same matrix to the embedding? It doesn't make sense to me