Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from $P_{1}, \dots, P_{k}$ and some data from $Q$ and we aim to find which distribution $P_{1}, \dots, P_{k}$ is the closest to $Q$ is a sense of KL-divergence.

KL-divergence, $D_{KL}(P_{i}||Q) = \int_{-\infty}^{\infty}p(x)\log\left(\frac{p(x)}{q(x)}\right)dx \neq D_{KL}(Q||P_{i})$, is not symmetric.

Therefore, if we compare $Q$ to all $P_{i}$, which one $D_{KL}(P_{i}||Q)$ or $D_{KL}(Q||P_{i})$, for $i = 1, \dots, k$ is correct to consider as the criterion?

From what I know, in AIK criterion one goes for $D_{KL}(Q||P_{i})$ case.


My confusion is partly from the following fact that KL is a premetric, it generates a topology on the space of probability distributions. Let us consider the sequence of measures $U_{1}, \dots, U_{n}$. Then if $$ \lim_{i\to\infty}D_{KL}(U_{i}||Q) = 0 $$ then $$ U_{n} \xrightarrow{d} Q. $$

  • 1
    $\begingroup$ Vis-a-vis the edit, I do not see why convergence in distribution implies the KL divergence goes to zero in either mode. The distributions $U_n$ and $Q$ can be singular with respect to each other and still have convergence in distribution, but not in KL. $\endgroup$ – guy Sep 24 '20 at 16:58
  • $\begingroup$ hm... I am also confused.. well, I think the opposite is correct, i.e. from second statement in UPDATE follows to first one. Do you agree? $\endgroup$ – ABK Sep 24 '20 at 21:03
  • 1
    $\begingroup$ Yes, I agree. KL convergence implies convergence in total variation by Pinsker's inequality, which implies convergence in distribution by definition (total variation implies the expectation of any bounded measurable function converges to the correct limit, while convergence in distribution only asks this of bounded continuous functions). $\endgroup$ – guy Sep 24 '20 at 22:00
  • $\begingroup$ Thank you. I made changes. $\endgroup$ – ABK Sep 24 '20 at 23:33

In $$\DeclareMathOperator{\E}{\mathbb{E}} D_{KL}(P || Q) = \int_{-\infty}^{\infty}p(x)\log\left(\frac{p(x)}{q(x)}\right)\;dx = \E_{P}\log\left(\frac{p(X)}{q(X)}\right) $$ we see this is the expectation of the loglikelihood ratio when $P$ is the truth, see Intuition on the Kullback-Leibler (KL) Divergence.

If, in hypothesis test language, $P$ is the null while $Q$ is the alternative: So $D_{KL}(P || Q)$ is divergence of $Q$ from (null) truth, while $D_{KL}(Q || P)$ is divergence when the alternative hypothesis is taken as truth. Then your question:

which distribution $P_1,\dotsc,P_k$ is the closest to $Q$ is a sense of KL-divergence?

If this means you want a model which is difficult to distinguish from $Q$ when/if $Q$ is the truth, you needs $D_{KL}(Q || P)$. Remember, the first argument is the truth (This is a way of saying that we calculate the divergence calculating an expectation assuming that the distribution generating $X$ is the distribution given in the first argument. That is, the truth about what is generating $X$.)

  • $\begingroup$ This is actually the opposite of the correct answer. Leaving aside the comments about null hypothesis equals truth, the interpretation of the KL divergence is exactly opposite of what this comment says. The linked post is correct though. $\endgroup$ – MentatOfDune Jan 7 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.