I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).

  • $\begingroup$ Why does it need to be symmetric, given that one of the two arguments is fixed anyway? A benefit of KL is that it’s easy to compute… $\endgroup$ Mar 25, 2022 at 2:24
  • $\begingroup$ what about the triangle inequality? Isn't it important to satisfy that? $\endgroup$
    – Sam
    Mar 25, 2022 at 3:02
  • 1
    $\begingroup$ @Zzy1130 the vast majority of variational problems have a true distribution and an approximating distribution. The KL divergence $K(p,q)$ permits an interpretation of the expected excess surprise when using $q$ as the working model although $p$ is the true model. So symmetry or triangle inequality wouldn’t make sense. $\endgroup$
    – Daeyoung
    Mar 25, 2022 at 4:46
  • 3
    $\begingroup$ A valid criticism of variational inference is that it uses $K(q,p)$ instead, which is the wrong direction in view of the above interpretation. But it’s a necessary trade off to make computation tractable. Otherwise, more assumptions are needed $\endgroup$
    – Daeyoung
    Mar 25, 2022 at 4:48
  • $\begingroup$ I think our combined point is: why do you think these properties are needed? Or is it just because you’re more familiar with distances than divergences? $\endgroup$ Mar 25, 2022 at 10:44


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.