I have a generative process as follows:

$$ x \mid \alpha \sim \textsf{Beta}\left (\alpha,\beta \right) \\ y \mid x \sim \textsf{Bernoulli}(x). $$

How does one go about calculating the Entropy of this process? Do we consider the beta-binomial (with $n=1$) instead?

Not quite sure where to start on this one, suggestions are most welcome. Thx.

Update 1

I believe now that the correct approach is to take the Beta-Binomial PMF (with $n=1$):

$$ P(k \mid 1,\alpha ,\beta )= {1 \choose k}{\frac {{\mathrm {B}}(k+\alpha ,1-k+\beta )}{{\mathrm {B}}(\alpha ,\beta )}}\! $$ where $\text{B}(\cdot)$ is the Beta function. This PMF can also be written as:

$$ P(k \mid 1,\alpha ,\beta )={\frac {\Gamma (1+1)}{\Gamma (k+1)\Gamma (1-k+1)}}{\frac {\Gamma (k+\alpha )\Gamma (1-k+\beta )}{\Gamma (1+\alpha +\beta )}}{\frac {\Gamma (\alpha +\beta )}{\Gamma (\alpha )\Gamma (\beta )}}. $$

and substitute it into the Shannon entropy:

$$ {\displaystyle \mathrm {H} (X)=\sum _{i=1}^{n}{\mathrm {P} (x_{i})\,\mathrm {I} (x_{i})}=-\sum _{i=1}^{n}{\mathrm {P} (x_{i})\log _{b}\mathrm {P} (x_{i})}.} $$

Update 2

Here is how far I have got. But first, lets remind ourselves of the model:

$$ X\sim \operatorname {Bin} (n,p) $$ then $$ P(X=k \mid p,n)=L(p|k)={n \choose k}p^{k}(1-p)^{n-k} $$ with $n=1$ we get $$ P(X=k \mid p,1)=L(p \mid k)={1 \choose k}p^{k}(1-p)^{1-k} $$ so we are saying that $X$ is defined on a binary space $\{0,1 \}$ also $$ {\binom {n}{k}}={\frac {n!}{k!(n-k)!}} = /n=1 / = {\binom {1}{k}}{\frac {1!}{k!(1-k)!}} $$

Recall also that entropy is defined as:

$$ \mathrm{H} (X) =\mathbb {E} [-\log(\mathrm {P} (X))] $$ Lets plug in our PMF expression (defined in update 1) for the Beta-Binomial: $$ \mathrm{H} [k] = \mathbb{E} \left [ - \log{\left (\frac{{\binom{1}{k}}}{\mathrm{B}{\left (\alpha,\beta \right )}} \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} \right )} \right] $$ which simplifies to $$ \begin{align} \mathrm{H} [k] &= \mathbb{E} \left [ \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )} - \log{{\binom{1}{k}}} \right ] \\ &= \mathbb{E}\left [\log{\mathrm{B}{\left (\alpha,\beta \right )}}\right ] - \mathbb{E} \left[\log \mathrm{B}{\left (\alpha + k,\beta - k + 1 \right )}\right ] - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{align} $$

Which reduces to:

$$ \begin{equation} \mathrm{H} [k] = \log{\mathrm{B}{\left (\alpha,\beta \right )}} - \psi(\alpha+k) + \psi(\alpha + \beta + 1) - \mathbb{E} \left [\log{{\binom{1}{k}}} \right]. \end{equation} $$

where $\psi(\cdot)$ is the digamma function. The problem is now the last expectation:

$$ \mathbb{E} \left [\log{{\binom{1}{k}}} \right] $$

Not sure if this makes sen; how can one take the expectation of a binomial coefficient? I feel like I have gone wrong somewhere.


Do we consider the beta-binomial (with $n=1$) instead?

Yes. The beta-binomial distribution is exactly the compound distribution of a binomial r.v. where the probability of success is, itself, a beta deviate. A Bernoulli r.v. is exactly a binomial r.v. with $n=1$.

  • $\begingroup$ Ah great! I must say I am somewhat confused though. Presumably, the entropy of the beta-binomial must be a standard result? But my googling is coming up with nothing. I found this (section 6): arxiv.org/pdf/1708.06394.pdf - but it is from 2017. Suggestions for good resources? $\endgroup$
    – Astrid
    May 2 '18 at 9:39
  • $\begingroup$ I suppose I could just substitute the compound PMF of the beta-binomial, into the Shannon entropy, and then simply calculate the entropy? Following this example: math.stackexchange.com/questions/394957/… and using the PMF from Wikipedia: en.wikipedia.org/wiki/Beta-binomial_distribution $\endgroup$
    – Astrid
    May 2 '18 at 9:52
  • $\begingroup$ I'm afraid I don't have any resources that specifically address how to compute the beta-bernoulli or beta-binomial entropy. It certainly appears that you're on the right track though: working forward from the definition of entropy. $\endgroup$
    – Sycorax
    May 2 '18 at 14:56
  • $\begingroup$ Not a problem, thanks for your help; your initial suggestion was very useful. $\endgroup$
    – Astrid
    May 2 '18 at 14:58

I'm not sure why you're not happy with the result in "update 1". To be explicit for provenance, consider the compound distribution

$$ \begin{aligned} X &\sim \text{Bernoulli}(p), \\ p &\sim \text{beta}(\alpha, \beta). \end{aligned} $$

If we integrate out $p$, the density is beta-Bernoulli,

$$ p(k \mid, \alpha, \beta) = \frac{\text{B}(k + \alpha, 1 - k + \beta)}{\text{B}(\alpha, \beta)}. $$

The entropy is

$$ \begin{aligned} \mathbb{H}[X \mid \alpha, \beta] &= -\sum_{x \in \{0,1\}} p(x \mid \alpha, \beta) \log p(x \mid \alpha, \beta) \\ &= - \left[ \frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(\alpha, 1 + \beta)}{\text{B}(\alpha, \beta)}\right) + \frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)} \log\left(\frac{\text{B}(1 + \alpha, \beta)}{\text{B}(\alpha, \beta)}\right)\right]. \end{aligned} $$

In many of your equations, you seem to not rely on the fact that $x$ has finite support over just $\{0,1\}$. Also, values like ${1 \choose k}$ are just $1$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.