Timeline for An adaptation of the Kullback-Leibler distance?
Current License: CC BY-SA 2.5
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 7, 2011 at 14:59 | comment | added | whuber♦ | @Didier Thank you: I misread the formula for $\eta$ until just this moment; I had confused it with the $\delta$ defined in another answer. Too hasty. I appreciate your patient explanations as well as the clarifications provided by @cardinal in comments to your answer. | |
| Feb 7, 2011 at 14:23 | comment | added | Did | @whuber Here is another example to let you better understand the behaviour of $\eta$. Let $U_t$ denote the uniform distribution on the interval $(t,1+t)$. Then $\eta(U_t,U_s)=\min(1,|t-s|)2\log(2)$. No $U_t$ and $U_s$ are absolutely continuous with respect to each other unless $s=t$. Finally, note that $\eta\le2\log(2)$ uniformly, and that $\eta(P,Q)=2\log(2)$ iff $P$ and $Q$ are mutually singular. | |
| Feb 7, 2011 at 14:09 | comment | added | Did | @whuber Why is there a concern about the possibility of infinite distance? Dunno... maybe because infinite distances is the feature the OP asked a remedy for in the first place... Re $\eta$, let me stress once again that $\eta(P,Q)$ is finite for every probability measures $P$ and $Q$. | |
| Feb 7, 2011 at 13:58 | comment | added | whuber♦ | @Didier Could you clarify what you mean by "finitized"? After all, two distributions can be infinitely far apart w.r.t. $\eta$, even when their supports overlap. The family of uniforms you discuss is special in that in each pair, one of them is absolutely continuous w.r.t the other. (Even that doesn't in general guarantee $\eta$ is finite, but it suffices in this case.) However, we are moving off onto a tangential topic here: what we are missing is any sense of why there is a concern about the possibility of an infinite divergence (or infinite distance). | |
| Feb 7, 2011 at 13:53 | comment | added | Did | .../... Example: for every $t>0$, let $P_t$ denote the uniform probability distribution on the interval $(0,t)$. For every $t\ne s$, $P_t$ and $P_s$ are at infinite distance for KL but $\eta(P_t,P_s)$ is positive and finite and depends non trivially on $(t,s)$. Inter alia, $\eta$ quantifies the fact that, if $t<s<u$, then $P_t$ is closer to $P_s$ than to $P_u$. I understood this is what Marco was after but maybe I am mistaken. | |
| Feb 7, 2011 at 13:52 | comment | added | Did | @whuber Topology and the (defect of) triangle inequality are not my point here. I simply wondered why Marco suddenly seemed happy with the $\arctan$ artefact you suggested, although this artefact leaves every pair of not absolutely continuous probability measures as infinitely far far away from each other as they were with respect to KL divergence. Note also that $\eta$ is not only (and not mainly) a symmetrized version of KL, more importantly $\eta$ is finitized. .../... | |
| Feb 7, 2011 at 13:22 | comment | added | whuber♦ | @Didier Yes, the transformed KL divergence (when symmetrized, as you describe) might not satisfy the triangle inequality and therefore would not be a distance, but it would still define a topology (which would likely be metrizable). You would thereby give up little or nothing. I remain agnostic about the merits of doing any of this: it seems to me this is just a way of papering over the difficulties associated with infinite values of the KL divergence in the first place. | |
| Feb 7, 2011 at 8:47 | comment | added | Did | @Marco: I am lost. Do you settle for the Kolmogorov distance (which is always finite but has nothing in common with KL divergence)? Or for a bounded monotone transform of KL divergence (such as $\arctan$)? In the example of your post (and in any other not absolutely continuous example), the latter produces the supremum of the transform ($\pi/2$ if you settle for $\arctan$). In effect, this abandons any idea of estimating a distance between such probability measures more precisely than saying they are far far away (whether you encode this by $\pi/2$ or by $+\infty$ is irrelevant). | |
| Feb 7, 2011 at 7:29 | vote | accept | ocram | ||
| Feb 9, 2011 at 10:57 | |||||
| Feb 7, 2011 at 7:17 | comment | added | ocram | Yes, that's what I meant :-) I was not sure on what to apply the transformation. Now, it is clear, thx | |
| Feb 6, 2011 at 19:12 | comment | added | whuber♦ | @Marco I don't understand how one could be any more explicit. Do you mean restating what I wrote in terms of a formula such as $\arctan(KL(P,Q))$ or $f(KL(P,Q))$ for $f:\mathbb{R_+} \to [0,C]$ with $x \ge y$ implies $f(x) \ge f(y)$ for all $x,y \ge 0$? | |
| Feb 6, 2011 at 16:52 | vote | accept | ocram | ||
| Feb 6, 2011 at 16:52 | |||||
| Feb 6, 2011 at 8:13 | comment | added | ocram | Thank you for your suggestion about the Kolmogorov distance. Can you make your comment about the monotonic transformation a little bit more explicit? Thx | |
| Feb 5, 2011 at 19:26 | history | answered | whuber♦ | CC BY-SA 2.5 |