I'm trying to get my head around the KL divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see the "difference in information" when $H_1$ is used instead of the better suited $H_0$ to describe the data. I write the data likelihood for each datum as $p(X_i \mid H_0)$ and $p(X_i \mid H_1)$. I could write the observed KL divergence as $$ \sum_{i=1}^N p(X_i \mid H_0) \log \frac{p(X_i \mid H_0)}{p(X_i \mid H_1)} $$ However, both likelihoods over the sample data $X$ do not sum to one. Therefore, I'm not sure what the interpretation would be. I do consider the two situations (a) where I compare a good hypothesis $H_0$ to a number of suboptimal alternatives and (b) where I compare two independent pairs $(H_0, H_1)$ and $(H'_0, H'_1)$. All comparisons are done over the same sample data $X$.

  • $\begingroup$ See stats.stackexchange.com/questions/188903/… $\endgroup$ – kjetil b halvorsen Aug 3 '17 at 13:48
  • $\begingroup$ Why do you expect likelihoods to sum to one over the data? $\endgroup$ – kjetil b halvorsen Aug 6 '17 at 20:40
  • $\begingroup$ Reading the other thread, this is not about symmetry. The question here was about comparability of KL divergences between different hypotheses (a) nested and (b) unrelated. It's about the unit of the KL divergence and normalization. $\endgroup$ – fungs Feb 11 '19 at 6:45
  • $\begingroup$ Well, the linked thread is not (only) about symmetry, for instance see myanswer there. If you after rereading still finds it does not answer your question, then edit this question and explain why/how the linked thread do not help you, then we will reopen. $\endgroup$ – kjetil b halvorsen Feb 11 '19 at 7:36