All Questions
Tagged with transformers tensor
3 questions
3
votes
1
answer
457
views
Difference between multi-head and single-head attention
Attention, as long as gradient calculations care, is two nested tensor multiplications and a softmax. I thought that, then, multi-head attention with $h=8$ and $d_k=64$ results in the same tensor with ...
- 1,099
1
vote
1
answer
170
views
Transformers: Cross Attention Tensor Shapes During Inference Mode
Using the "classic" transformer model describing in "Attention is All You Need", I'm struggling to understand how the Encoder output is used by the Decoder during cross attention ...
- 156
0
votes
1
answer
402
views
bos_token for a custom Transformer
I am trying to use a Transformer to solve a time-series problem. I built the model using the Pytorch library. And I am planning to train the model from scratch. The model is looking back last L time-...
- 3,459