0
$\begingroup$

I'm following this paper on ICA and I got to equation (1) describing the multivariate mutual information contrast function as a sum of entropies.

$J(Y) = \int p(y_1,...y_D)log(\frac{p(y_1,...y_D)}{p(y_1)p(y_2)...p(y_D))})du$

where $du = dy_1dy_2...dy_D$

$ = KL(p(y_1,...y_D)\parallel\displaystyle\prod_{i=1}^{D}p(y_i))$

$ = \displaystyle\sum_{i=1}^{D}H(Y_i) - H(Y_1,...Y_D)$

This seems to be in line with this post referencing a metric called "total correlation". I'm curious as to where the first summation term comes from.

From the first line, I can see how $J(Y)$ could be written as

$J(Y) = -\int p(y_1,...y_D)log(p(y_1)p(y_2)...p(y_D))du + \int p(y_1,...y_D)log(p(y_1,...y_D))du$

so the second term appears to be joint entropy

$-H(Y_1,...Y_D) = \int p(y_1,...y_D)log(p(y_1,...y_D))du$

The first term is what I'd like clarification on.

Does

$\displaystyle\sum_{i=1}^{D}H(Y_i) = -\int p(y_1,...y_D)log(p(y_1)p(y_2)...p(y_D))du$

because

$-\displaystyle\int p(y_1,...y_D)log(p(y_1)p(y_2)...p(y_D))du = \displaystyle\int_{y_1 \in Y_1}...\displaystyle\int_{y_D \in Y_D} p(y_1,...y_D)[log(p(y_1)) + ... + log(p(y_D))]dy_1...dy_D $

and for each term every other $y$ is marginalized out? i.e. :

$\displaystyle\int_{y_1 \in Y_1}...\displaystyle\int_{y_D \in Y_D} p(y_1,...y_D)[log(p(y_n))]dy_1...dy_D = \displaystyle\int_{y_n \in Y_n} p(y_n)log(p(y_n))dy_n = H(Y_n)$

$ where, n \in 1...D$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.