3
$\begingroup$

Let $X\sim \operatorname{Pois}(\lambda)$ and $x_1,\ldots,x_n$ observations following this distribution. I want to derive the analytical solution of the following series: $$\ell(\lambda):=\lim_{x\rightarrow \infty} \frac{1}{n}\sum_{i=1}^n \log P(X=x_i).$$

EDIT: After a few trials, I found a good numerical approximation of the solution: $$\ell(\lambda)=-\frac{1}{2}\log(17.08\cdot \lambda).$$ See the graph below, where the dots represent an approximation of the solution by simulating poisson distributions, while the blue line represents the approximated numerical solution.

x=1:1000
y=sapply(x,function(x) mean(log(dpois(rpois(100000,x),x))))
plot(x,y)
lines(x,-log(sqrt(x*17.08)),col="blue")

Numerical solution

$\endgroup$
3
  • $\begingroup$ That's right on the mark: the constant is $2\pi e \approx 17.07947.$ See the formula for entropy at en.wikipedia.org/wiki/Poisson_distribution. As a check, follow up with curve(-(log(2*pi*exp(1)*x)/2 - 1/(12*x) - 1/(24*x^2) - 19/(360*x^3)), add=TRUE, col="Red", lwd=2, n=501) as suggested by the formula at that page. BTW, for faster iteration you don't need such a tightly spaced set of $\lambda.$ Use, say, x=exp(seq(0, 20, by=1/2)) and include log="x" as an argument to plot. $\endgroup$
    – whuber
    Feb 2, 2019 at 16:52
  • $\begingroup$ Thanks whuber, I didn't know that what I wanted to calculate was actually the entropy! If you had added an answer instead of commenting my question, I could have selected it as the right answer. $\endgroup$ Feb 3, 2019 at 16:45
  • $\begingroup$ Your use of the word "serie", with no final "s" makes me wonder if you're more accustomed to French than to English. Or was that just a typo? $\qquad$ $\endgroup$ Feb 3, 2019 at 19:15

1 Answer 1

3
$\begingroup$

If you have $n$ samples of a random variables $Y$ (or equivalently, $n$ iid random variables) and you compute the sample mean as

$$ m_n = \frac{1}{n} \sum_{i=1}^n Y_i$$

then, as $n$ grows, (under certain conditions) $m_n$ will converge (in some sense) to the expected value of $Y$, that is $m_n \to E[Y]$. See the law of large numbers, which clarifies those "certain conditions" and in what sense the "convergence" occurs.

In our case, $Y=\log(p(X))$ hence

$$\begin{align} m_n \to E[\log(p(X)]&=\sum_x p_X(x) \log(p_X(x)) \\ &= \sum_{x=0}^\infty e^{-\lambda} \frac{\lambda^x}{x!} (-\lambda + x \log(\lambda) - \log(x!)) \\ &= -\lambda (1 - \log(\lambda)) - e^{-\lambda} \sum_{x=0}^\infty\frac{\lambda^x}{x!}\log(x!) \end{align}$$

This cannot be put in simpler form, but for large $\lambda$ it can be shown that it tends to $ -\frac{1}{2} \log(2 \pi e \lambda)$

Except for the sign, as noted in a comment, this is the entropy of a Poisson distribution, see eg here or the Wikipedia.

BTW, if you look for higher order correction terms, bear in mind that most results in the bibliography imply a base two logarithm (as usual in information theory).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.