I have a generative process as follows:

$$ x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\ y \mid x \sim \textsf{Bernoulli}(x). $$

How does one go about calculating the Entropy of this process? Do we consider the beta-binomial (with $n=1$) instead?

Not quite sure where to start on this one, suggestions are most welcome. Thx.

Update 1

I believe now that the correct approach is to take the Beta-Binomial PMF (with $n=1$):

$$ P(k \mid 1,\alpha ,\beta )= {1 \choose k}{\frac {{\mathrm {B}}(k+\alpha ,1-k+\beta )}{{\mathrm {B}}(\alpha ,\beta )}}\! $$ where $\text{B}(\cdot)$ is the Beta function. This PMF can also be written as:

$$ P(k \mid 1,\alpha ,\beta )={\frac {\Gamma (1+1)}{\Gamma (k+1)\Gamma (1-k+1)}}{\frac {\Gamma (k+\alpha )\Gamma (1-k+\beta )}{\Gamma (1+\alpha +\beta )}}{\frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}}. $$

and substitute it into the Shannon entropy:

$$ {\displaystyle \mathrm {H} (X)=\sum _{i=1}^{n}{\mathrm {P} (x_{i})\,\mathrm {I} (x_{i})}=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}.} $$

Update 2

Here is how far I have got. But first, lets remind ourselves of the model:

$$ X\sim \operatorname {Bin} (n,p) $$ then $$ P(X=k \mid p,n)=L(p|k)={n \choose k}p^{k}(1-p)^{n-k} $$ with $n=1$ we get $$ P(X=k \mid p,1)=L(p \mid k)={1 \choose k}p^{k}(1-p)^{1-k} $$ so we are saying that $X$ is defined on a binary space $\{0,1 \}$ also $$ {\binom {n}{k}}={\frac {n!}{k!(n-k)!}} = /n=1 / = {\binom {1}{k}}{\frac {1!}{k!(1-k)!}} $$

Recall also that entropy is defined as:

$$ \mathrm{H} (X) =\mathbb {E} [-\log(\mathrm {P} (X))] $$ Lets plug in our PMF expression (defined in update 1) for the Beta-Binomial: $$ \mathrm{H} [k] = \mathbb{E} \left [ - \log{\left (\frac{{\binom{1}{k}}}{\mathrm{B}{\left (\alpha,\beta \right )}} \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} \right )} \right] $$ which simplifies to $$ \begin{align} \mathrm{H} [k] &= \mathbb{E} \left [ \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} - \log{{\binom{1}{k}}} \right ] \\ &= \mathbb{E}\left [\log{\mathrm{B}{\left (\alpha,\beta \right )}}\right ] - \mathbb{E} \left[\log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )}\right ] - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{align} $$

Which reduces to:

$$ \begin{equation} \mathrm{H} [k] = \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \psi(\alpha+k) + \psi(\alpha + \beta + 1) - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{equation} $$

where $\psi(\cdot)$ is the digamma function. The problem is now the last expectation:

$$ \mathbb{E} \left [\log{{\binom{1}{k}}} \right] $$

Not sure if this makes sen; how can one take the expectation of a binomial coefficient? I feel like I have gone wrong somewhere.


3 Answers 3


For the sake of completeness since this question continues to appear highly in search results, you're absolutely on the right track with the calculations in update 1. However, you can simplify this greatly using the fact that you have a Bernoulli random variable conditioned on the draw from a beta distribution, and not a general binomial random variable.

The pmf for a Beta-Binomial distribution is given by

$$P(k; \alpha, \beta, n) = {n \choose k}\frac{\text{B}(\alpha+k, \beta + n - k)}{\text{B}(\alpha,\beta)}$$

But with the simplification of $n = 1$ this reduces to

$$\frac{\alpha^k \beta^{1-k}}{\alpha+\beta}$$

Because of the following:

  1. ${1 \choose k} = 1$ for $k \in {0,1}$
  2. $\text{B}(x, y) = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$
  3. $\Gamma(x+1) = x \Gamma(x)$ and therefore $\frac{\Gamma(x+1)}{\Gamma(x)} = x$

(1) allows dropping the binomial coefficient. Expanding the beta functions into products of gamma functions with (2) results in, for $k=1$, $$\frac{\Gamma(\alpha+1)\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\alpha+\beta+1)}$$

And (3) can be used to rewrite that as $\frac{\alpha}{\alpha + \beta}$. The probability of $k=0$ is of course one minus the above, and that's everything needed for calculating the entropy.

$$H(k) = -P(k=1)\log(P(k=1))-P(k=0)\log(P(k=0)) = \log(\alpha+\beta) - \frac{1}{\alpha+\beta}(\alpha \log(\alpha) + \beta \log(\beta))$$

Note that this is simply the entropy of Bernoulli random variable with $p$ equal to the expected value of the beta distribution. This is not true in general (i.e. for $n \gt 1$) but this special case is simple because a Bernoulli random variable with a beta success probability is indistinguishable from a Bernoulli random variable with the success probability equal to the mean of the beta.

Also note that this is different question from "what is the expected entropy of a Bernoulli random variable when the success probability is drawn from a beta distribution?" In that case, we're looking $\int f(p) (-p\log(p) -(1-p)\log(1-p))$ which is a fairly straightforward calculation (hint: separate into the sum of two integrals, and find constants that can be multiplied inside/divided outside the integral to cancel overall but make the integral look like $E[\log(p)]$ etc with altered parameters).

Finally, the difficulty you're having with a closed form expression for the entropy of the Beta-Binomial distribution (without summing over k) in general is insurmountable- there is no closed form expectation for the entropy of the binomial distribution alone for $n\gt1$ and the situation just gets worse with the compound distribution. Typically, I like to start with the more general case, solve the problem, and then simplify, but unfortunately your problem does not have a nice solution unless applying the special case first.


Do we consider the beta-binomial (with $n=1$) instead?

Yes. The beta-binomial distribution is exactly the compound distribution of a binomial r.v. where the probability of success is, itself, a beta deviate. A Bernoulli r.v. is exactly a binomial r.v. with $n=1$.

  • $\begingroup$ Ah great! I must say I am somewhat confused though. Presumably, the entropy of the beta-binomial must be a standard result? But my googling is coming up with nothing. I found this (section 6): arxiv.org/pdf/1708.06394.pdf - but it is from 2017. Suggestions for good resources? $\endgroup$
    – Astrid
    May 2, 2018 at 9:39
  • $\begingroup$ I suppose I could just substitute the compound PMF of the beta-binomial, into the Shannon entropy, and then simply calculate the entropy? Following this example: math.stackexchange.com/questions/394957/… and using the PMF from Wikipedia: en.wikipedia.org/wiki/Beta-binomial_distribution $\endgroup$
    – Astrid
    May 2, 2018 at 9:52
  • $\begingroup$ I'm afraid I don't have any resources that specifically address how to compute the beta-bernoulli or beta-binomial entropy. It certainly appears that you're on the right track though: working forward from the definition of entropy. $\endgroup$
    – Sycorax
    May 2, 2018 at 14:56
  • $\begingroup$ Not a problem, thanks for your help; your initial suggestion was very useful. $\endgroup$
    – Astrid
    May 2, 2018 at 14:58

I'm not sure why you're not happy with the result in "update 1". To be explicit for provenance, consider the compound distribution

$$ \begin{aligned} X &\sim \text{Bernoulli}(p), \\ p &\sim \text{beta}(\alpha, \beta). \end{aligned} $$

If we integrate out $p$, the density is beta-Bernoulli,

$$ p(k \mid, \alpha, \beta) = \frac{\text{B}(k + \alpha, 1 - k + \beta)}{\text{B}(\alpha, \beta)}. $$

The entropy is

$$ \begin{aligned} \mathbb{H}[X \mid \alpha, \beta] &= -\sum_{x \in \{0,1\}} p(x \mid \alpha, \beta) \log p(x \mid \alpha, \beta) \\ &= - \left[ \frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)}\right) + \frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)}\right)\right]. \end{aligned} $$

In many of your equations, you seem to not rely on the fact that $x$ has finite support over just $\{0,1\}$. Also, values like ${1 \choose k}$ are just $1$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.