KL-Divergence and Entropy for marginals I am going through this paper, where the following claim is unproven (page 3, after the first equality):
Let $r,c \in \mathbb{R}_+^d$ be discrete probability histograms, and $P \in \mathbb{R}_+^{d,d}$ a joint probability matrix that has $r,c$ as marginals. Then we have
$$KL(P||rc^T) = H(r) + H(c)-H(P).$$
It is supposed to be easy, but I dont see how I can arrive at this result.
 A: With $r,c$, being $d\times 1$ column vectors, their outer product results in the matrix $rc^T$. We want to compare this resulting $d\times d$ distribution with any $P$ that has $r,c$ as marginals. I let subscript $i,j$ indicate the position in the matrices, and by $[d] = \{1,2,\ldots, d-1, d\}$. Thus,
$$KL(P||rc^T) = \sum_{(i,j) \in [d]\times[d]} P_{i,j} \log \frac{P_{i,j}}{rc^T_{i,j}} = \sum_{(i,j) \in [d]\times[d]} P_{i,j} \log P_{i,j} - P_{i,j} \log rc^T_{i,j}. $$
Now for any $i$, when you sum over each $j \in [0,d]$, you marginalize out the dependence on $j$, and similarly for fixed $j$ and varying $i$. But we also know that $P_{i,j}$ has the correct marginals of $r$ and $c$. Thus for the second term, you get
\begin{align}
\sum_{(i,j) \in [d]\times[d]}  - P_{i,j} \log rc^T_{i,j} &= \sum_{(i,j) \in [d]\times[d]}  - P_{i,j} \log r_i - P_{i,j} \log c_j 
\\&= -\sum_i \log r_i \sum_{j} P_{i,j} - \sum_j \log c_j \sum_{i} P_{i,j} \\&= -\sum_{i} r_i \log r_i - \sum_{j} c_j \log c_j,
\end{align}
and by the definition of entropy $H(p) = -\sum_{i} p_{i} \log p_{i}$, we get the result. 
