1
$\begingroup$

I'm trying to re-implement Transformer on my own but I'm struggling on the input encoding.

It seems that the original paper(Attention is All You Need) mentioned that they use Byte-pair Encoding to encode words into vectors but as I understand BPE is just used for word segmentation (compress size of the dictionary) so how will it produce a numeric vector?

In many implementations, I saw that they just initialled random weights for word embedding. In this case, will the weights be updated when the model is being trained?

Lastly, As they mentioned that "we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation", I understand that two embedding layers could be shared but why "the pre-softmax linear transformation" should be the same matrix to the embedding? It doesn't make sense to me

$\endgroup$
1
$\begingroup$

as I understand, BPE is just used for word segmentation (compress size of the dictionary) so how will it produce a numeric vector?

BPE(an illustration can be found here) (and unigram language model) is a subword algorithm, and it just tokenizes the text into subwords which can be just treated as words separated by spaces because they will be transformed into ordinal ids by a mapping dictionary or (feature) hashing.

The embedding matrix will be trainable(when you pretrain the model) or untrainable(in some conditions for instance transfer learning afterwards for some downstream tasks).

I understand that two embedding layers could be shared but why "the pre-softmax linear transformation" should be the same matrix to the embedding? It doesn't make sense to me

In this paper: Using the Output Embedding to Improve Language Models the author found that sharing the parameters would benefit the performance. The author also found that in Word2vec the $W'$, which is also an embedding matrix, is kind of inferior to the standard embedding matrix $W$. Some guys tried to tie these two matrices but that leads to the detriment. But in large neural network language models, such parameter sharing is reportedly benefitial.

enter image description here

Source: https://stackoverflow.com/a/42392407/3552975

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.