Kullback-Leibler distance calculation for discrete distributions?

Question

I have the following model

$$N \sim Pois(\lambda) \\ n \sim Bin(N,p)$$

for which I calculate the posterior for the parameter $N$ as

$$\pi(N|n,p,\lambda) = \frac{f(n|N,p)\pi(N|\lambda)}{f(n|p,\lambda)}$$

My goal is to calculate the Kullback-Leibler distance between the posterior and prior, i.e. $KL = \sum \pi(N|n,p)log\frac{\pi(N|n,p,\lambda)}{\pi(N|\lambda)}$

---

$1)$The first step is to calculate the prior predictive $f(n|p,\lambda)$

$f(n|p,\lambda) = \sum_{N=0}^{\infty} \binom{N}{n}p^{n}(1-p)^{N-n}\frac{\lambda^{N}e^{-\lambda}}{N!}$

However, for $N<n$ I believe the $\textit{Binomial}$ distribution would be meaningless, so I start the sum from $N=n,n+1,...

$f(n|p,\lambda) = \sum_{N=n}^{\infty} \binom{N}{n}p^{n}(1-p)^{N-n}\frac{\lambda^{N}e^{-\lambda}}{N!}$

Because, I do not know how to calculate it analytically, I calculate it numerically by generating a sequence of numbers $n,n+1,..., M$, where $M$ is a sufficiently large number that will make the $\textit{Poisson}$ prior zero (so I truncate the summation), and finally iteratively calculate the sum.

$2)$ Now that I have calculated (hope correctly) the prior predictive and I know all the terms of the posterior as functions of $N$, I can move forward to calculating numerically the $KL$ distance.

$$KL = \sum_{N=n}^{\infty} \pi(N|n,p)log\frac{\pi(N|n,p,\lambda)}{\pi(N|\lambda)}$$

I again simulate a large sequence of numbers $n,n+1,..., M$ for a sufficiently large $M$.

---

Questions:

$i)$ Is there a way to calculate analytically the prior predictive $f(n|p,\lambda)$? Or what I did is used in practice??

$ii)$ In the second step, there would be a problem when $N=M$ (or when $N$ starts getting unlikely values) because the $\textit{Poisson}$ prior would be zero, something that has to be avoided because it is on the denominator. Is there a way to alleviate that problem?

$iii)$ The ranges we conduct the summations are correct?? I thought that they should be different because in the prior predictive we take the sum over the range of prior beliefs (however, we take into account the value of $n$ which is somehow counterintuitive). Whereas in the $KL$ the summation should be over the values of the posterior range, I assume they coincide?

Any suggestions would be great!

Answer 1

To start, yes, you're right, the Binomial distribution doesn't make sense for $N < n$. More precisely, we can say that $f(n | N, p) = 0$ for $N < n$ (intuitively, you're saying that the probability of getting $n$ successes in $N$ tries is zero). You can then write your summation as you did, starting from $N = n$. Then, you can analytically compute the prior predictive distribution as follows:

$p(n | p, \lambda) = \sum_{N=n}^{\infty} f(n | N, p) \pi(N|\lambda) = \sum_{N=n}^{\infty} {N \choose n} p^n (1-p)^{N-n} \frac{\lambda^N}{N!} e^{-\lambda}$

$= \frac{(\lambda p)^n}{e^\lambda} \sum_{N=n}^\infty \frac{N!}{n! (N-n)!} \frac{(\lambda(1-p))^{N-n}}{N!} = \frac{(\lambda p)^n}{n!e^\lambda} \sum_{N=n}^\infty \frac{(\lambda(1-p))^{N-n}}{(N-n)!}$

$= \frac{(\lambda p)^n}{n!e^\lambda} \sum_{k=0}^\infty \frac{(\lambda(1-p))^{k}}{k!} = \frac{(\lambda p)^n}{n!e^\lambda} e^{\lambda(1-p)} = \frac{(\lambda p)^n}{n!} e^{-\lambda p}$

You can see that this is again a Poisson distribution, with parameter $\lambda p$. Note that this is similar to a conjugate prior, but a rare one, because usually in a Binomial likelihood $N$ is considered known and $p$ is the target of statistical inference.

I hope this helps answer your questions!