All Questions
Tagged with transformers machine-translation
10 questions
8
votes
1
answer
573
views
Why are Transformers "suboptimal" for language modeling but not for translation?
Language Models with Transformers states:
Transformer architectures are suboptimal for language model itself. Neither self-attention nor the positional encoding in the Transformer is able to ...
- 1,347
3
votes
1
answer
1k
views
Why is Bahdanau's attention sometimes called concat attention?
I am learning the intuition behind the attention mechanism from
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/
https://lilianweng.github....
- 165
3
votes
1
answer
2k
views
How does inference work on a Transformer?
(Let's say I trained a transformer for translation.)
In the training, the output sentences are given and fed into the decoder as a whole. However, with inference, only a start-of-sentence (SOS) token ...
2
votes
1
answer
59
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 395
2
votes
1
answer
113
views
What feature space is used in transformer networks for machine translation?
Title is the question.
The papers I've read, e.g. "Attention is All You Need" fail to specify exactly what word embeddings are used in these machine translation networks. In most cases ...
- 3,269
1
vote
1
answer
254
views
Why do we feed the target sentence to the model during transformers training?
Going through the famous notebook: Annotated transformers to be found here, I noticed that during the training, the model is fed the target sentence.
If the model is fed the target sentence from the ...
- 111
1
vote
1
answer
489
views
Why are K and V the same in the second attention layer of a Transformer's decoder?
(For this example, let's say we're using a Transformer to translate from English to French.)
For the decoder in a Transformer, the second attention layer takes K and V from the encoder and then takes ...
1
vote
1
answer
1k
views
How does Transformer use BPE?
I'm trying to re-implement Transformer on my own but I'm struggling on the input encoding.
It seems that the original paper(Attention is All You Need) mentioned that they use Byte-pair Encoding to ...
- 135
0
votes
1
answer
700
views
Number of weights to be learnt in the encoder decoder attention in the transformer model
I have a doubt about the number of weights to be learned in the encoder-decoder attention layer in the transformer model (attention is all you need). Some blogs articles say the $K$ and $V$ (key and ...
- 55
0
votes
0
answers
400
views
Training and testing transformer model from scratch
As you know, transformers are one of the strongest model in the field of NLP and machine translation.
I know there are many resources, but I still could not find a good tutorial teaching how to use ...
- 395