I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-attention layer doesn't learn this relationship between words anyways, making the embedding layer seem redundant to me.
I guess I'm mostly confused on how learning $W_Q,W_K,W_V$ fails to capture what's learned in the embedding weights. Is it something to do with the difference in how the embedding layer and self-attention layer are trained?
If this has to do with the difference between contextual and non-contextual word embeddings, then my question stems to if contextual embeddings already contain the information that non-contextual embeddings have?
Is it at all possible to just skip learning the embedding layer when mapping the vocabulary to tokens as input to the self-attention layer?
