Suppose I have two distributions over the probability simplex $P_S$ and $Q_S$, so that e.g. $ p\sim P_S $ and $q\sim Q_S$ are vectors such that $\sum_i p_i=\sum_i q_i =1$ with $p_i,q_i\geq 0\;\forall \;i$ (i.e. $p$ and $q$ can be viewed as parametrizing two different categorical distributions).

Given independent $ p\sim P_S $ and $q\sim Q_S$, I can have an "estimated" cross-entropy: $$ \widehat{H}[p,q] = -\sum_i p_i\log q_i $$ Is it true that $\widehat{H}$ estimates the (continuous) cross-entropy between $P_S$ and $Q_S$ in an unbiased, consistent way?

Even if true in a special case, say logit-normal, it would be interesting to me.

For instance, does $\mathbb{E}[\widehat{H}[p,q]]=H[P_S,Q_S]$?

Here's some attempt from me: let $\mathbb{E}[p]=\mu$, then $$ \mathbb{E}\left[\widehat{H}[p,q]\right] = -\sum_i \mathbb{E}[p_i]\mathbb{E}[\log q_i] = -\sum_i\mu_i\mathbb{E}[\log q_i] $$ but it's not clear how this leads anywhere useful. Maybe using the fact that $ H[p,q] = D_{\text{KL}}[p,q] + H[p] $ could be helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.