# Entropy of a set of categorical variables

In the context of Expectation-Maximization, I would like to compute te entropy factor in order to get the value of the lower bound when the algorithm converged.

This lower bound can be expressed as:

$$\mathcal{L}(q, \theta) = \underbrace{\sum_{\mathbf{Z}} p(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta}^{old})\ln p(\mathbf{X,Z|\boldsymbol{\theta}})}_{\mathcal{Q}(\boldsymbol{\theta},\boldsymbol{\theta}^{old})}- \underbrace{\sum_{\mathbf{Z}} p(\mathbf{Z}|\mathbf{X}, \boldsymbol{\theta}^{old})\ln p(\mathbf{Z} | \mathbf{X}, \boldsymbol{\theta}^{old})}_\mathcal{H}$$

(notation from Bishop's Pattern Recognition and Machine Learning, page 452)

In the E-M we ignore the entropy and focus on $\mathcal{Q}$ because the entropy does not depend on $\boldsymbol{\theta}$. But once we finish, if we want to compare different models, we need the total $\mathcal{L}$ and thus we need to compute the entropy $\mathcal{H}$ (if my understanding is correct).

I'm seeing some strange behaviors in my computed $\mathcal{L}$, and I'm starting to think that I'm not computing entropy correctly.

So, I have $n$ points and their probabilities $\mathbf{z}_1,...,\mathbf{z}_n$. Each probability distribution represents the probability of belonging to each of the components (responsibilities) . If we have two components:

$$p(\mathbf{z}_i | \cdot ) = (z_{i1}, z_{i2})$$

How should I compute the entropy?

I'm currently computing this:

\begin{align} p(z_{11}) \ln p(z_{11}) &+ p(z_{12}) \ln p(z_{12}) ~+\\ p(z_{21}) \ln p(z_{21}) &+ p(z_{22}) \ln p(z_{22}) ~+\\ ...\\ p(z_{n1}) \ln p(z_{n1}) &+ p(z_{n2}) \ln p(z_{n2}) \end{align}

• Were you able to find anything out about this over the last 25 days? – Robert Sep 26 '16 at 5:02
• That the entropy of two independent variables is the sum of the entropies, and thus my way of computing it was correct. I still would like a third opinion to confirm that everything is fine. – alberto Sep 27 '16 at 15:23
• I asked my professor about this, and he said that Renyi Entropy (en.wikipedia.org/wiki/R%C3%A9nyi_entropy) is a generalization of Shannon Entropy that you suggested here. Although it's not a third option, it might help! – Robert Sep 27 '16 at 16:44