Let $X\sim \operatorname{Pois}(\lambda)$ and $x_1,\ldots,x_n$ observations following this distribution. I want to derive the analytical solution of the following series: $$\ell(\lambda):=\lim_{x\rightarrow \infty} \frac{1}{n}\sum_{i=1}^n \log P(X=x_i).$$

EDIT: After a few trials, I found a good numerical approximation of the solution: $$\ell(\lambda)=-\frac{1}{2}\log(17.08\cdot \lambda).$$ See the graph below, where the dots represent an approximation of the solution by simulating poisson distributions, while the blue line represents the approximated numerical solution.

y=sapply(x,function(x) mean(log(dpois(rpois(100000,x),x))))

Numerical solution

  • $\begingroup$ That's right on the mark: the constant is $2\pi e \approx 17.07947.$ See the formula for entropy at en.wikipedia.org/wiki/Poisson_distribution. As a check, follow up with curve(-(log(2*pi*exp(1)*x)/2 - 1/(12*x) - 1/(24*x^2) - 19/(360*x^3)), add=TRUE, col="Red", lwd=2, n=501) as suggested by the formula at that page. BTW, for faster iteration you don't need such a tightly spaced set of $\lambda.$ Use, say, x=exp(seq(0, 20, by=1/2)) and include log="x" as an argument to plot. $\endgroup$ – whuber Feb 2 at 16:52
  • $\begingroup$ Thanks whuber, I didn't know that what I wanted to calculate was actually the entropy! If you had added an answer instead of commenting my question, I could have selected it as the right answer. $\endgroup$ – Anthony Hauser Feb 3 at 16:45
  • $\begingroup$ Your use of the word "serie", with no final "s" makes me wonder if you're more accustomed to French than to English. Or was that just a typo? $\qquad$ $\endgroup$ – Michael Hardy Feb 3 at 19:15

If you have $n$ samples of a random variables $Y$ (or equivalently, $n$ iid random variables) and you compute the sample mean as

$$ m_n = \frac{1}{n} \sum_{i=1}^n Y_i$$

then, as $n$ grows, (under certain conditions) $m_n$ will converge (in some sense) to the expected value of $Y$, that is $m_n \to E[Y]$. See the law of large numbers, which clarifies those "certain conditions" and in what sense the "convergence" occurs.

In our case, $Y=\log(p(X))$ hence

$$\begin{align} m_n \to E[\log(p(X)]&=\sum_x p_X(x) \log(p_X(x)) \\ &= \sum_{x=0}^\infty e^{-\lambda} \frac{\lambda^x}{x!} (-\lambda + x \log(\lambda) - \log(x!)) \\ &= -\lambda (1 - \log(\lambda)) - e^{-\lambda} \sum_{x=0}^\infty\frac{\lambda^x}{x!}\log(x!) \end{align}$$

This cannot be put in simpler form, but for large $\lambda$ it can be shown that it tends to $ -\frac{1}{2} \log(2 \pi e \lambda)$

Except for the sign, as noted in a comment, this is the entropy of a Poisson distribution, see eg here or the Wikipedia.

BTW, if you look for higher order correction terms, bear in mind that most results in the bibliography imply a base two logarithm (as usual in information theory).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.