Entropy of the beta-binomial compound distribution

Question

I have a generative process as follows:

$$ x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\ y \mid x \sim \textsf{Bernoulli}(x). $$

How does one go about calculating the Entropy of this process? Do we consider the beta-binomial (with $n=1$) instead?

Not quite sure where to start on this one, suggestions are most welcome. Thx.

Update 1

I believe now that the correct approach is to take the Beta-Binomial PMF (with $n=1$):

$$ P(k \mid 1,\alpha ,\beta )= {1 \choose k}{\frac {{\mathrm {B}}(k+\alpha ,1-k+\beta )}{{\mathrm {B}}(\alpha ,\beta )}}\! $$ where $\text{B}(\cdot)$ is the Beta function. This PMF can also be written as:

$$ P(k \mid 1,\alpha ,\beta )={\frac {\Gamma (1+1)}{\Gamma (k+1)\Gamma (1-k+1)}}{\frac {\Gamma (k+\alpha )\Gamma (1-k+\beta )}{\Gamma (1+\alpha +\beta )}}{\frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}}. $$

and substitute it into the Shannon entropy:

$$ {\displaystyle \mathrm {H} (X)=\sum _{i=1}^{n}{\mathrm {P} (x_{i})\,\mathrm {I} (x_{i})}=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}.} $$

Update 2

Here is how far I have got. But first, lets remind ourselves of the model:

$$ X\sim \operatorname {Bin} (n,p) $$ then $$ P(X=k \mid p,n)=L(p|k)={n \choose k}p^{k}(1-p)^{n-k} $$ with $n=1$ we get $$ P(X=k \mid p,1)=L(p \mid k)={1 \choose k}p^{k}(1-p)^{1-k} $$ so we are saying that $X$ is defined on a binary space $\{0,1 \}$ also $$ {\binom {n}{k}}={\frac {n!}{k!(n-k)!}} = /n=1 / = {\binom {1}{k}}{\frac {1!}{k!(1-k)!}} $$

Recall also that entropy is defined as:

$$ \mathrm{H} (X) =\mathbb {E} [-\log(\mathrm {P} (X))] $$ Lets plug in our PMF expression (defined in update 1) for the Beta-Binomial: $$ \mathrm{H} [k] = \mathbb{E} \left [ - \log{\left (\frac{{\binom{1}{k}}}{\mathrm{B}{\left (\alpha,\beta \right )}} \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} \right )} \right] $$ which simplifies to $$ \begin{align} \mathrm{H} [k] &= \mathbb{E} \left [ \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} - \log{{\binom{1}{k}}} \right ] \\ &= \mathbb{E}\left [\log{\mathrm{B}{\left (\alpha,\beta \right )}}\right ] - \mathbb{E} \left[\log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )}\right ] - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{align} $$

Which reduces to:

$$ \begin{equation} \mathrm{H} [k] = \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \psi(\alpha+k) + \psi(\alpha + \beta + 1) - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{equation} $$

where $\psi(\cdot)$ is the digamma function. The problem is now the last expectation:

$$ \mathbb{E} \left [\log{{\binom{1}{k}}} \right] $$

Not sure if this makes sen; how can one take the expectation of a binomial coefficient? I feel like I have gone wrong somewhere.

Answer 1

For the sake of completeness since this question continues to appear highly in search results, you're absolutely on the right track with the calculations in update 1. However, you can simplify this greatly using the fact that you have a Bernoulli random variable conditioned on the draw from a beta distribution, and not a general binomial random variable.

The pmf for a Beta-Binomial distribution is given by

$$P(k; \alpha, \beta, n) = {n \choose k}\frac{\text{B}(\alpha+k, \beta + n - k)}{\text{B}(\alpha,\beta)}$$

But with the simplification of $n = 1$ this reduces to

$$\frac{\alpha^k \beta^{1-k}}{\alpha+\beta}$$

Because of the following:

1. ${1 \choose k} = 1$ for $k \in {0,1}$
2. $\text{B}(x, y) = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$
3. $\Gamma(x+1) = x \Gamma(x)$ and therefore $\frac{\Gamma(x+1)}{\Gamma(x)} = x$

(1) allows dropping the binomial coefficient. Expanding the beta functions into products of gamma functions with (2) results in, for $k=1$, $$\frac{\Gamma(\alpha+1)\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\alpha+\beta+1)}$$

And (3) can be used to rewrite that as $\frac{\alpha}{\alpha + \beta}$. The probability of $k=0$ is of course one minus the above, and that's everything needed for calculating the entropy.

$$H(k) = -P(k=1)\log(P(k=1))-P(k=0)\log(P(k=0)) = \log(\alpha+\beta) - \frac{1}{\alpha+\beta}(\alpha \log(\alpha) + \beta \log(\beta))$$

Note that this is simply the entropy of Bernoulli random variable with $p$ equal to the expected value of the beta distribution. This is not true in general (i.e. for $n \gt 1$) but this special case is simple because a Bernoulli random variable with a beta success probability is indistinguishable from a Bernoulli random variable with the success probability equal to the mean of the beta.

Also note that this is different question from "what is the expected entropy of a Bernoulli random variable when the success probability is drawn from a beta distribution?" In that case, we're looking $\int f(p) (-p\log(p) -(1-p)\log(1-p))$ which is a fairly straightforward calculation (hint: separate into the sum of two integrals, and find constants that can be multiplied inside/divided outside the integral to cancel overall but make the integral look like $E[\log(p)]$ etc with altered parameters).

Finally, the difficulty you're having with a closed form expression for the entropy of the Beta-Binomial distribution (without summing over k) in general is insurmountable- there is no closed form expectation for the entropy of the binomial distribution alone for $n\gt1$ and the situation just gets worse with the compound distribution. Typically, I like to start with the more general case, solve the problem, and then simplify, but unfortunately your problem does not have a nice solution unless applying the special case first.

Answer 2

Do we consider the beta-binomial (with $n=1$) instead?

Yes. The beta-binomial distribution is exactly the compound distribution of a binomial r.v. where the probability of success is, itself, a beta deviate. A Bernoulli r.v. is exactly a binomial r.v. with $n=1$.

Answer 3

I'm not sure why you're not happy with the result in "update 1". To be explicit for provenance, consider the compound distribution

$$ \begin{aligned} X &\sim \text{Bernoulli}(p), \\ p &\sim \text{beta}(\alpha, \beta). \end{aligned} $$

If we integrate out $p$, the density is beta-Bernoulli,

$$ p(k \mid, \alpha, \beta) = \frac{\text{B}(k + \alpha, 1 - k + \beta)}{\text{B}(\alpha, \beta)}. $$

The entropy is

$$ \begin{aligned} \mathbb{H}[X \mid \alpha, \beta] &= -\sum_{x \in \{0,1\}} p(x \mid \alpha, \beta) \log p(x \mid \alpha, \beta) \\ &= - \left[ \frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)}\right) + \frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)}\right)\right]. \end{aligned} $$

In many of your equations, you seem to not rely on the fact that $x$ has finite support over just $\{0,1\}$. Also, values like ${1 \choose k}$ are just $1$.