# Attention Mechanisms in Tranformers, effect of using more parameters than Keys and Queries

In the attention mechanism, as described in the paper "Attention is all you need", normally the model learns weights $$W_Q$$ and $$W_K$$, i.e. the Queries and the Keys. The input of the attention layer is multiplied with both weights to get the Queries and Keys

$$$$Q = W_Q^TX~\text{and}~K = W_K^TX$$$$

before computing the attention weights, called $$a_w$$ in this context, using the softmax function:

$$$$a_w = \operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)$$$$

where $$\sqrt{d_k}$$ is the square root of the dimension of the input. To my current understanding, and described in very newbie terms, the idea of using different weights for the Queries and Keys, is to project the input to different spaces, to potentially find useful connections between each element in the input. (A useful connection corresponds to a high softmax entry.)

My question is now, how important actually is the aspect of "different spaces"? What if we, in theory, without computational issues, use $$4$$ weight matrices? What if we only use $$1$$ weight matrix? That is, compute the keys and queries like so:

$$\begin{array}{cc} Q = W_Q^TX & K = W_Q^TX \end{array}$$

Wouldn't we still be able to find useful connections for each input?

The answer to your question is yes: you'd find useful connections still. Setting $$W_Q \triangleq W_K$$ is one of the suggestions for parameter efficiency from Kitaev et al. (2020 at ICLR). It has the effect of making the queries and keys identical.