My goal is to estimate the entropy of a multinomial distribution, based on a single observation (a set of counts for each possible outcome). I also want to calculate the uncertainty in my estimated entropy based on the uncertainty in the pmf. I'm confused about the case of two outcomes with equal probability.

With $N$ total counts and outcomes X and Y, the variance/covariance matrix for the counts is: $$ \sigma_X^2 = \sigma_Y^2 = N/4 \\ \sigma_{XY} = -N/4. $$ The covariance for the observed frequencies is then: $$ \sigma_X^2 = \sigma_Y^2 = 1/(4N) \\ \sigma_{XY} = -1/(4N). $$ Now I apply the variance formula to the entropy calculation: $$ S = -\sum_i p_i \lg(p_i) = -p_X\lg(p_X) - (1-p_X)\lg(1-p_X)\\ \frac{\partial S}{\partial p_X} = \lg\left( \frac{1-p_x}{p_x} \right)\\ \frac{\partial S}{\partial p_Y} = \lg\left( \frac{1-p_y}{p_y}\right)\\ \sigma_S^2 = \sigma_X^2\left|\frac{\partial S}{\partial p_X}\right|^2 + \sigma_Y^2\left|\frac{\partial S}{\partial p_Y}\right|^2 + 2\sigma_{XY} \left(\frac{\partial S}{\partial p_X}\right) \left(\frac{\partial S}{\partial p_Y}\right) $$

First question: Since $p_X$ fully constrains $p_Y$, should I be considering both in calculating $\sigma_S^2$?

Second question: when $p_X = p_Y = 1/2$, the partial derivatives are zero, which gives $\sigma_S^2 = 0$, which can't be right. What am I missing?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.