Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from $P_{1}, \dots, P_{k}$ and some data from $Q$ and we aim to find which distribution $P_{1}, \dots, P_{k}$ is the closest to $Q$ is a sense of KL-divergence.

KL-divergence, $D_{KL}(P_{i}||Q) = \int_{-\infty}^{\infty}p(x)\log\left(\frac{p(x)}{q(x)}\right)dx \neq D_{KL}(Q||P_{i})$, is not symmetric.

Therefore, if we compare $Q$ to all $P_{i}$, which one $D_{KL}(P_{i}||Q)$ or $D_{KL}(Q||P_{i})$, for $i = 1, \dots, k$ is correct to consider as the criterion?

From what I know, in AIK criterion one goes for $D_{KL}(Q||P_{i})$ case.


My confusion is partly from the following fact that KL is a premetric, it generates a topology on the space of probability distributions. Let us consider the sequence of measures $U_{1}, \dots, U_{n}$. Then if $$ \lim_{i\to\infty}D_{KL}(U_{i}||Q) = 0 $$ then $$ U_{n} \xrightarrow{d} Q. $$

  • 1
    $\begingroup$ Vis-a-vis the edit, I do not see why convergence in distribution implies the KL divergence goes to zero in either mode. The distributions $U_n$ and $Q$ can be singular with respect to each other and still have convergence in distribution, but not in KL. $\endgroup$
    – guy
    Sep 24, 2020 at 16:58
  • $\begingroup$ hm... I am also confused.. well, I think the opposite is correct, i.e. from second statement in UPDATE follows to first one. Do you agree? $\endgroup$
    – ABK
    Sep 24, 2020 at 21:03
  • 1
    $\begingroup$ Yes, I agree. KL convergence implies convergence in total variation by Pinsker's inequality, which implies convergence in distribution by definition (total variation implies the expectation of any bounded measurable function converges to the correct limit, while convergence in distribution only asks this of bounded continuous functions). $\endgroup$
    – guy
    Sep 24, 2020 at 22:00
  • $\begingroup$ Thank you. I made changes. $\endgroup$
    – ABK
    Sep 24, 2020 at 23:33

1 Answer 1


In $$\DeclareMathOperator{\E}{\mathbb{E}} D_{KL}(P || Q) = \int_{-\infty}^{\infty}p(x)\log\left(\frac{p(x)}{q(x)}\right)\;dx = \E_{P}\log\left(\frac{p(X)}{q(X)}\right) $$ we see this is the expectation of the loglikelihood ratio when $P$ is the truth, see Intuition on the Kullback-Leibler (KL) Divergence.

If, in hypothesis test language, $P$ is the null while $Q$ is the alternative: So $D_{KL}(P || Q)$ is divergence of $Q$ from (null) truth, while $D_{KL}(Q || P)$ is divergence when the alternative hypothesis is taken as truth. Then your question:

which distribution $P_1,\dotsc,P_k$ is the closest to $Q$ is a sense of KL-divergence?

If this means you want a model which is difficult to distinguish from $Q$ when/if $Q$ is the truth, you needs $D_{KL}(Q || P)$. Remember, the first argument is the truth (This is a way of saying that we calculate the divergence calculating an expectation assuming that the distribution generating $X$ is the distribution given in the first argument. That is, the truth about what is generating $X$.)

  • $\begingroup$ This is actually the opposite of the correct answer. Leaving aside the comments about null hypothesis equals truth, the interpretation of the KL divergence is exactly opposite of what this comment says. The linked post is correct though. $\endgroup$ Jan 7, 2021 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.