As in the title, should I expect a model to learn almost the same attention scores in its attention layers when I train it? Perhaps only in the first one if there are multiple such layers?
It feels intuitive to me that, if attention is learning the most 'important' features for the prediction, then it should learn almost the same scores, or at least give the same 'order' to the features.
Thank you!
The attention scores of a model directly depend on the weights of the model. Usually re-training a model (unless fixing the seed) will not lead to the same model weights, and for that reason, in many papers, you will see the same model being trained multiple times to see how the performance changes.
As the attention scores depend on the weights and the weights are usually not the same when training the same model again, the attention scores will also not be identical.
However, just because they are not identical that does not mean they are just random. If we properly train our model, we would expect the attention to be very similar especially if our training data is very large.
So: Attention Scores are not identical across multiple trainings of a model, but they should be very similar, given proper training of the models.
This is the general answer, however, there is an on-going debate, on whether attention really has something to do with "importance".
See for example:
https://arxiv.org/abs/2201.12114
People can manipulate attention, and hence in theory there are cases where similar models can have completely different attention scores.
