1
$\begingroup$

Given factorizations of two joint densities $p(x_1,...,x_n)=\prod_{i=1}^n p(x_i\mid \textrm{cond}(x_i))$ and $q(x_1,...,x_n)=\prod_{i=1}^n q(x_i\mid \textrm{cond}(x_i))$, where $\textrm{cond}(\bullet)$ denotes the set of conditioning variables, does the KL-divergence decompose, i.e., does

$\textrm{KL}(p\Vert q)= \sum_{i=1}^n \textrm{KL}\left(p(x_i\mid \textrm{cond}(x_i))\ \Vert\ q(x_i\mid \textrm{cond}(x_i))\right)$

hold?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ If it makes things easier, you can assume $\textrm{cond}(x_i) \in \{x_1,\ldots,x_n\}$ . More generally, the two factorisations are Bayesian networks, which means that $\textrm{cond}(x_i)$ can be any subset of $\{x_1,\ldots,x_n\}$ so that the induced graph structure is a directed acyclic graph. $\endgroup$ – ASML Jan 23 '16 at 14:23
  • 1
    $\begingroup$ How do you define $\textrm{KL}\left(p(x_i\mid \textrm{cond}(x_i))\ \Vert\ q(x_i\mid \textrm{cond}(x_i))\right)$ ? The ambiguous part is how you integrate out the $cond(x_i)$ variables in there $\endgroup$ – Guillaume Dehaene Feb 3 '16 at 13:42
0
$\begingroup$

As ASML suggested you can represent the factorization of the joint probability distribution according to a Bayesian network. Then, as it is pointed in [Tong, S., & Koller, D. (2001)] (page 4) the KL-divergence decomposes with the graphical structure of the network.

[Tong, S., & Koller, D. (2001)] Tong, S., & Koller, D., Active learning for parameter estimation in Bayesian networks. In Advances in neural information processing systems (pp. 647-653).

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ please add full reference instead of a link which may die $\endgroup$ – Antoine Mar 27 '18 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.