All Questions
13 questions
13
votes
1
answer
3k
views
When calculating self-attention for Transformer ML architectures, why do we need both a key and a query weight matrix?
I'm trying to understand the math behind Transformers, specifically self-attention. This link, and many others, gives the formula to compute the output vectors from the input embeddings as:
$$Q=XW_Q,\;...
- 133
9
votes
2
answers
4k
views
Why transformer in deep learning is called transformer?
Where does the name "transformer" come from in deep learning? I want to know more about the correlation between its name and its working principle.
- 343
5
votes
2
answers
10k
views
Self Attention for Variable Length Sequence Classification
I have a problem that is not particularly unique, but I'm still having trouble to figure out exactly how it's usually done.
My training set is of the form $\mathcal{T}=\{(t_i\in \mathbb{R}^{[n,m]\...
- 160
4
votes
1
answer
4k
views
Training Transformers: self attention weights vs embedding layer
I have been trying to wrap my head around transformers. While I have found many good resources that explain the self attention mechanism I've yet to find a good answer on how it really works with ...
- 143
3
votes
1
answer
65
views
Operation modes in neural turing machine (Graves, 2014)
I am reading the paper "Neural Turing Machines" of Alex Graves (2014) and there are two points that are unclear to me. I would be very grateful if someone could help me out.
More ...
- 77
2
votes
1
answer
59
views
In cases where neural attention is used for machine translation, how they deal with translating sentences that have different lengths?
So attention and transformer models can be used for machine translation.
Sometimes, a sentence in one language might consist of 5 words, but in the target language it consists of 8 words (so for ...
- 395
2
votes
0
answers
29
views
Contrast Self attention and Attention in machine learning for non technical person?
How would you contrast Self attention and Attention in machine learning for a non technical person?
I really like this explanation of Attention from Chatgpt: When someone speaks to you, you naturally ...
- 1,021
1
vote
1
answer
729
views
How does a transformer network's attention mechanism determine what to pay attention to?
The paper Attention is All You Need describes the transformer network which uses a "multi-head attention mechanism". The basic intuition behind this mechanism is that it weighs other tokens ...
1
vote
1
answer
1k
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
3
answers
2k
views
transformer, linear layer after attention
My question regards this image:
It seems that after the multi head attention there is a linear layer as they mention also from here:
the linearity is given by the weights W^{o}. my quesion is: for ...
- 13
0
votes
1
answer
40
views
Does a model learn the same attention scores when retrained?
As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers?
It feels ...
- 11
0
votes
0
answers
35
views
Why learn an embedding before self attention when training transformers?
I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
0
votes
1
answer
108
views
Dimension Mismatch in Transformer Decoder: What Are the Input and Output Dimensions?
My understanding is that in the decoder, the output of the masked self-attention mechanism is expected to have dimensions (o_len,d_model), where o_len is the current output length.
However, an issue ...
- 1