Suppose I know the true distribution, $P^*(x)$ and I have approximated the true distribution with $\tilde{P}(x|D)$, which is the predictive posterior density. Does the relative entropy of $\tilde{P}(x|D)$ with respect to $P^*(x)$ become: $$D_{KL}(P^{*}||\tilde{P})=\int_{-\infty}^{\infty}P^{*}(x)\,\log\frac{P^*(x)}{\tilde{P}(x|D)} dx$$
Also, does a smaller relative entropy in this case also imply a smaller distance between my hypothesis and the true distribution?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Yes, by definition of the Kullback-Leibler divergence.
Yes, since $D_{KL}(P^{*}||\tilde{P})=0$ if and only if $P^{*}=\tilde{P}$ almost surely.
-
$\begingroup$ @mackbox You may also want to consider other distances such as the Total Variation distance, which is the L1 distance in unidimensional models $\int\vert P^*(x)-\tilde{P}(x)\vert dx$. $\endgroup$– KassioAug 26, 2016 at 15:38