# Entropy of the beta-binomial compound distribution

I have a generative process as follows:

$$x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\ y \mid x \sim \textsf{Bernoulli}(x).$$

How does one go about calculating the Entropy of this process? Do we consider the beta-binomial (with $n=1$) instead?

Not quite sure where to start on this one, suggestions are most welcome. Thx.

# Update 1

I believe now that the correct approach is to take the Beta-Binomial PMF (with $n=1$):

$$P(k \mid 1,\alpha ,\beta )= {1 \choose k}{\frac {{\mathrm {B}}(k+\alpha ,1-k+\beta )}{{\mathrm {B}}(\alpha ,\beta )}}\!$$ where $\text{B}(\cdot)$ is the Beta function. This PMF can also be written as:

$$P(k \mid 1,\alpha ,\beta )={\frac {\Gamma (1+1)}{\Gamma (k+1)\Gamma (1-k+1)}}{\frac {\Gamma (k+\alpha )\Gamma (1-k+\beta )}{\Gamma (1+\alpha +\beta )}}{\frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}}.$$

and substitute it into the Shannon entropy:

$$\mathrm {H} (X)=\sum _{i=1}^{n}{\mathrm {P} (x_{i})\,\mathrm {I} (x_{i})}=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}.}$$

# Update 2

Here is how far I have got. But first, lets remind ourselves of the model:

$$X\sim \operatorname {Bin} (n,p)$$ then $$P(X=k \mid p,n)=L(p|k)={n \choose k}p^{k}(1-p)^{n-k}$$ with $n=1$ we get $$P(X=k \mid p,1)=L(p \mid k)={1 \choose k}p^{k}(1-p)^{1-k}$$ so we are saying that $X$ is defined on a binary space $\{0,1 \}$ also $${\binom {n}{k}}={\frac {n!}{k!(n-k)!}} = /n=1 / = {\binom {1}{k}}{\frac {1!}{k!(1-k)!}}$$

Recall also that entropy is defined as:

$$\mathrm{H} (X) =\mathbb {E} [-\log(\mathrm {P} (X))]$$ Lets plug in our PMF expression (defined in update 1) for the Beta-Binomial: $$\mathrm{H} [k] = \mathbb{E} \left [ - \log{\left (\frac{{\binom{1}{k}}}{\mathrm{B}{\left (\alpha,\beta \right )}} \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} \right )} \right]$$ which simplifies to \begin{align} \mathrm{H} [k] &= \mathbb{E} \left [ \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} - \log{{\binom{1}{k}}} \right ] \\ &= \mathbb{E}\left [\log{\mathrm{B}{\left (\alpha,\beta \right )}}\right ] - \mathbb{E} \left[\log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )}\right ] - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{align}

Which reduces to:

$$\begin{equation} \mathrm{H} [k] = \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \psi(\alpha+k) + \psi(\alpha + \beta + 1) - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{equation}$$

where $\psi(\cdot)$ is the digamma function. The problem is now the last expectation:

$$\mathbb{E} \left [\log{{\binom{1}{k}}} \right]$$

Not sure if this makes sen; how can one take the expectation of a binomial coefficient? I feel like I have gone wrong somewhere.

## 1 Answer

Do we consider the beta-binomial (with $n=1$) instead?

Yes. The beta-binomial distribution is exactly the compound distribution of a binomial r.v. where the probability of success is, itself, a beta deviate. A Bernoulli r.v. is exactly a binomial r.v. with $n=1$.

• Ah great! I must say I am somewhat confused though. Presumably, the entropy of the beta-binomial must be a standard result? But my googling is coming up with nothing. I found this (section 6): arxiv.org/pdf/1708.06394.pdf - but it is from 2017. Suggestions for good resources? – Astrid May 2 '18 at 9:39
• I suppose I could just substitute the compound PMF of the beta-binomial, into the Shannon entropy, and then simply calculate the entropy? Following this example: math.stackexchange.com/questions/394957/… and using the PMF from Wikipedia: en.wikipedia.org/wiki/Beta-binomial_distribution – Astrid May 2 '18 at 9:52
• I'm afraid I don't have any resources that specifically address how to compute the beta-bernoulli or beta-binomial entropy. It certainly appears that you're on the right track though: working forward from the definition of entropy. – Sycorax May 2 '18 at 14:56
• Not a problem, thanks for your help; your initial suggestion was very useful. – Astrid May 2 '18 at 14:58