All Questions
Tagged with machine-learning transformers
2 questions
13
votes
1
answer
3k
views
When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?
I'm trying to understand the math behind Transformers, specifically self-attention. This link, and many others, gives the formula to compute the output vectors from the input embeddings as:
$$Q=XW_Q,\;...
- 133
6
votes
2
answers
10k
views
What is the length limit of Transformers?
From hugginface documentation:
Transformer-based models are unable to process long sequences due to their self-attention operation
How long is "long" here? 1000 inputs? 10,000 inputs?
Just ...
- 331