I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
$\begingroup$
$\endgroup$
6
-
$\begingroup$ Why does it need to be symmetric, given that one of the two arguments is fixed anyway? A benefit of KL is that it’s easy to compute… $\endgroup$– Arya McCarthyCommented Mar 25, 2022 at 2:24
-
$\begingroup$ what about the triangle inequality? Isn't it important to satisfy that? $\endgroup$– SamCommented Mar 25, 2022 at 3:02
-
1$\begingroup$ @Zzy1130 the vast majority of variational problems have a true distribution and an approximating distribution. The KL divergence $K(p,q)$ permits an interpretation of the expected excess surprise when using $q$ as the working model although $p$ is the true model. So symmetry or triangle inequality wouldn’t make sense. $\endgroup$– DaeyoungCommented Mar 25, 2022 at 4:46
-
3$\begingroup$ A valid criticism of variational inference is that it uses $K(q,p)$ instead, which is the wrong direction in view of the above interpretation. But it’s a necessary trade off to make computation tractable. Otherwise, more assumptions are needed $\endgroup$– DaeyoungCommented Mar 25, 2022 at 4:48
-
$\begingroup$ I think our combined point is: why do you think these properties are needed? Or is it just because you’re more familiar with distances than divergences? $\endgroup$– Arya McCarthyCommented Mar 25, 2022 at 10:44
|
Show 1 more comment