I have been digging in the concept of entropy for a while, now it comes to the implementation part I feel I am confused.

Imagine that we have a matrix 20 * 3 standing for 20 words 3 topics (by 20 words I mean the probability of those words in docs, and the topics as the column is just like clusters).

So the problem here is related to topic modeling on the text data.

In the implementations, I have seen they apply entropy in each row: enter image description here

Also, I qoute a part of the paper:
"If the topic-word distribution
p(t|w) is uniform, the word is not characteristic of any
topic. If it is highly concentrated then it is. This can
be captured using the inverse of the entropy H(w):"

Now I am unable to interpret, what conclusion I can derive if I say have 20 entropy? how can I result that the cluster is distinctive while it is a sum over all topics!?

I would like to see each word is characteristic of which topics, what can I do for that?

and if it can help this is the link of paper paper, and in part 3. I appreciate if someone shed more light on the topic.


I would say, entropy measures the uniformity of a probability distribution, the higher the entropy, the uniformer the distribution. So if we calculate the entropy of each word (row), then the words with the lowest entropy are those that characterize their topic(s) most.

Calculating the entropy does not answer which topics are described by each word, only how "distinctive" each word is.

Maybe this paper is related to your question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.