# Does the Transformer decoder query based on the previous token?

Consider the decoder part of the popular Transformer architecture; briefly put, the decoder module consists of a composition of self-attention layers and performs auto-regressive prediction. Because of the masked attention, the output at any time step $$j$$ depends only on the previous tokens, from $$1$$ to $$j$$. Mathematically, a simplified one self-attention layer would look like:

$$\mathbf{y}_j = \mathrm{softmax}\left(\sum_{i=1}^j \left\langle \mathbf{k}(\mathbf{x}_i), \mathbf{q}(\mathbf{x}_j) \right\rangle \mathbf{v}(\mathbf{x}_i)\right)$$

However, in order for the model not to cheat and just copy the input to the output, the input sequence is shifted to the right with one token. So, the output $$\mathbf{y}_j$$ corresponds in fact to the prediction of the next token, $$\hat{\mathbf{x}}_{j+1}$$:

$$\hat{\mathbf{x}}_{j+1} = \mathrm{softmax}\left(\sum_{i=1}^j \left\langle \mathbf{k}(\mathbf{x}_i), \mathbf{q}(\mathbf{x}_j) \right\rangle \mathbf{v}(\mathbf{x}_i)\right)$$

If we consider a concrete example

the probability for the word "sat" given the previous words is

$$p(\mathrm{sat} | \mathsf{s}, \mathrm{the}, \mathrm{cat}) = \mathrm{softmax}\left(\alpha(\mathsf{s}, \mathrm{cat}) \mathbf{v}(\mathsf{s}) + \alpha(\mathrm{the}, \mathrm{cat}) \mathbf{v}(\mathrm{the}) + \alpha(\mathrm{cat},\mathrm{cat})\mathbf{v}(\mathrm{cat})\right)_{\mathrm{sat}}$$

where $$\mathsf{s}$$ and $$\mathsf{e}$$ denote the start and end tokens, and $$\alpha$$ the key-query inner product.

Given the above, it seems to me that the Transformer decoder pools the features based on the previous token, the $$j$$-th one, to predict the current one, the $$j +1$$-th.

1. Is my understanding correct?
2. What is the motivation for using the previous token as a way to aggregate the features for the current token? I could think of alternatives to using the previous token $$\mathbf{x}_j$$ to build the query: use the positional encoding $$\mathbf{p}_{j+1}$$ or average all the previous vectors $$\mathbf{x}_1, \dots, \mathbf{x}_j$$.