I’m currently working on building a Vision Transformer (ViT), and so far, things seem to be going well — low loss values, high accuracy. However, when I visualized the attention maps, I noticed that they fade over time and become uniform. I expected the opposite — that as the model learns, the attention maps would be used by the transformer to identify which patches have a greater influence on decision-making and which have less. Initially, this is indeed the case, but as the model continues learning, the attention maps become increasingly uniform.
It seems that either there’s something wrong with my model, or the transformer stops paying attention to the relationships between patches during decision-making. I’m curious if anyone else has encountered this behavior and can help me interpret what’s happening.
As for the model itself, it performs well and shows promising results, so I’m inclined to think there’s nothing wrong with it. However, it’s hard to say definitively what’s "right" or "wrong" in this case.
In short, I would appreciate any help in interpreting these results — I don’t understand why the attention maps become so uniform and their values negligible, indicating that the transformer may not be considering the relationships between patches in its decision-making process.
I use more then 10k samples. at 5 epoch i got this values Epoch 5/20 1179/1179 [==============================] - 1197s 1s/step - loss: 0.1253 - sparse_categorical_accuracy: 0.9950 - val_loss: 0.0607 - val_sparse_categorical_accuracy: 1.0000
thank for your help.
