# Expectation of Empirical Cross Entropy Estimator

Suppose I have two distributions over the probability simplex $P_S$ and $Q_S$, so that e.g. $p\sim P_S$ and $q\sim Q_S$ are vectors such that $\sum_i p_i=\sum_i q_i =1$ with $p_i,q_i\geq 0\;\forall \;i$ (i.e. $p$ and $q$ can be viewed as parametrizing two different categorical distributions).

Given independent $p\sim P_S$ and $q\sim Q_S$, I can have an "estimated" cross-entropy: $$\widehat{H}[p,q] = -\sum_i p_i\log q_i$$ Is it true that $\widehat{H}$ estimates the (continuous) cross-entropy between $P_S$ and $Q_S$ in an unbiased, consistent way?

Even if true in a special case, say logit-normal, it would be interesting to me.

For instance, does $\mathbb{E}[\widehat{H}[p,q]]=H[P_S,Q_S]$?

Here's some attempt from me: let $\mathbb{E}[p]=\mu$, then $$\mathbb{E}\left[\widehat{H}[p,q]\right] = -\sum_i \mathbb{E}[p_i]\mathbb{E}[\log q_i] = -\sum_i\mu_i\mathbb{E}[\log q_i]$$ but it's not clear how this leads anywhere useful. Maybe using the fact that $H[p,q] = D_{\text{KL}}[p,q] + H[p]$ could be helpful.