The Shannon entropy of terms for two documents $d_1$ and $d_2$ is computed

$$\displaystyle H_1(X) = -\sum_{i=1}^n P(t_i)\cdot \log_2(P(t_i))\quad\text{and similar for}\quad H_2(X).$$

  • Can the difference of two entropies in absolute value $\lvert H_1-H_2\rvert$ be used as a similarity metric?
  • If so, is this used anywhere?

( I wasn't able to find any context that uses this difference. Instead, I've seen the relative entropy being used for this. I was able to find an example of how it's computed here )

  • $\begingroup$ Using this similarity on its own is probably not useful, but as a feature in ML settings it will most likely be helpful $\endgroup$ – Alexey Grigorev Sep 3 '16 at 7:10
  • $\begingroup$ A log-likelihood distance is entropy-based. $\endgroup$ – ttnphns Sep 3 '16 at 12:12
  • $\begingroup$ @ttnphns sounds interesting, can you provide a link that describes it? preferably with an example of usage too $\endgroup$ – average Sep 4 '16 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.