1. Suppose I know the true distribution, $P^*(x)$ and I have approximated the true distribution with $\tilde{P}(x|D)$, which is the predictive posterior density. Does the relative entropy of $\tilde{P}(x|D)$ with respect to $P^*(x)$ become: $$D_{KL}(P^{*}||\tilde{P})=\int_{-\infty}^{\infty}P^{*}(x)\,\log\frac{P^*(x)}{\tilde{P}(x|D)} dx$$

  2. Also, does a smaller relative entropy in this case also imply a smaller distance between my hypothesis and the true distribution?


1 Answer 1

  1. Yes, by definition of the Kullback-Leibler divergence.

  2. Yes, since $D_{KL}(P^{*}||\tilde{P})=0$ if and only if $P^{*}=\tilde{P}$ almost surely.

  • $\begingroup$ @mackbox You may also want to consider other distances such as the Total Variation distance, which is the L1 distance in unidimensional models $\int\vert P^*(x)-\tilde{P}(x)\vert dx$. $\endgroup$
    – Kassio
    Aug 26, 2016 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.