This question concerns the application of self-attention weights for identifying influence of words in sentences.
For instance, we are performing a classification task on a set of sentences (e.g. sentiment analysis). The model takes the embedded words and feeds them into a (bidirectional) LSTM or GRU layer. The hidden states are combined by an attention vector to obtain a sentence representation that is used as an input for the classification.
For a set of sentences $S$, and the attention weights $\alpha^s = <\alpha^s_1, \alpha^s_2, \ldots, \alpha^s_n>$ for each sentence $s \in S$, we can combine attentions of all sentences into a matrix size $|W| \times |S|$ where $|W|$ is the vocabulary size and $|S|$ is the number of sentences, with column sum equal to 1 (each column is a probability distribution over words). On the other hand, sentence-class relationship forms a matrix of $|C| \times |S|$, where $|C|$ is the number of classes with each column a probability distribution over classes.
Using the above two learned probability distributions, is it possible to answer the following two questions:
1) For a particular class (e.g.,
excitement), what are the most influential words.
2) For a particular word, how much it influences a particular class. e.g. how much the word
Any pointers or references to similar work is highly appreciated.