1
$\begingroup$

Suppose I have two distributions over the probability simplex $P_S$ and $Q_S$, so that e.g. $ p\sim P_S $ and $q\sim Q_S$ are vectors such that $\sum_i p_i=\sum_i q_i =1$ with $p_i,q_i\geq 0\;\forall \;i$ (i.e. $p$ and $q$ can be viewed as parametrizing two different categorical distributions).

Given independent $ p\sim P_S $ and $q\sim Q_S$, I can have an "estimated" cross-entropy: $$ \widehat{H}[p,q] = -\sum_i p_i\log q_i $$ Is it true that $\widehat{H}$ estimates the (continuous) cross-entropy between $P_S$ and $Q_S$ in an unbiased, consistent way?

Even if true in a special case, say logit-normal, it would be interesting to me.

For instance, does $\mathbb{E}[\widehat{H}[p,q]]=H[P_S,Q_S]$?

Here's some attempt from me: let $\mathbb{E}[p]=\mu$, then $$ \mathbb{E}\left[\widehat{H}[p,q]\right] = -\sum_i \mathbb{E}[p_i]\mathbb{E}[\log q_i] = -\sum_i\mu_i\mathbb{E}[\log q_i] $$ but it's not clear how this leads anywhere useful. Maybe using the fact that $ H[p,q] = D_{\text{KL}}[p,q] + H[p] $ could be helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.