I'm using Graphlab's LDA functionality, and I get two matrices as the result: The document topic matrix (i.e.
P(Topic|Document) for each document) and the the topic-word matrix (i.e.
P(Word|Topic) for each topic).
What I want to accomplish is calculate the document-specific similarity between an arbitrary pair of words in a document, probably using Jensen Shannon divergence (though in principal, most any similarity measure could be used here).
Now, I can't just use rows of the word-topic matrix, since those really represent
P(Word|Topic), and to reasonably compute the similarity between two words I need
P(Topic|Word) (i.e. a probability distribution over topics for a given word). This is doable, I believe, with some renormalization of the matrices (basically by converting the
P(Word|Topic) values in the word-topic matrix to actual token frequencies, then normalizing word-wise instead of topic-wise). I'm pretty confident this works, but can add more detail if ther are questions. In any case, this captures the global (i.e. across the full corpus) similarity between any two words.
But now here's the kicker: Is there a principled way to calculate a document-specific topic distribution for a term that I could use for calculating the similarity between two terms within a particular document? To put it another way, instead of
P(Topic|Word), I need
The intuition this should capture is that if a document is focused on a particular subset of topics, there should be more meaningful distinctions between words that are strongly associated with those topics, whereas for words strongly associated with topics that are not important to the document, distinctions should be less strong. Here's a concrete (toy) example where our model has only 4 topics:
Let's say we have two words, a and b, with the following topic distributions (obtained using the method I described above):
In : a Out: array([ 0.9 , 0.05, 0.03, 0.02]) In : b Out: array([ 0.6 , 0.25, 0.1 , 0.05])
The distance (Jensen Shannon Divergence) between these would be:
In : JSD(a,b) Out: 0.064751544117297333
Now clearly both these words are skewed towards the first two topics, and let's now imagine a document that predominantly focuses on these topics, with the following topic distribution:
In : doc_x Out: array([ 0.7 , 0.2 , 0.05, 0.05])
and a second document that favors the other two topics:
In : doc_y Out: array([ 0.1, 0.1, 0.5, 0.5])
So based on all this, we should expect the distance between
b to be increased in
doc_x as compared to the global JSD, because that document focuses more on the first two topics, whereas in doc_y the distance should be decreased as compared to the global JSD. I believe this result should come out naturally if I use distributions of
P(Topic|Word,Document) instead of
Now, if I had the topic assignments for every token in the corpus (which in my case, is on the the order of 4 billion), this might be simpler, but I don't - just the two matrices above. Any ideas on how to go about this?