I have the following model

$$N \sim Pois(\lambda) \\ n \sim Bin(N,p)$$

for which I calculate the posterior for the parameter $N$ as

$$\pi(N|n,p,\lambda) = \frac{f(n|N,p)\pi(N|\lambda)}{f(n|p,\lambda)}$$

My goal is to calculate the Kullback-Leibler distance between the posterior and prior, i.e. $KL = \sum \pi(N|n,p)log\frac{\pi(N|n,p,\lambda)}{\pi(N|\lambda)}$

$1)$The first step is to calculate the prior predictive $f(n|p,\lambda)$

$f(n|p,\lambda) = \sum_{N=0}^{\infty} \binom{N}{n}p^{n}(1-p)^{N-n}\frac{\lambda^{N}e^{-\lambda}}{N!}$

However, for $N<n$ I believe the $\textit{Binomial}$ distribution would be meaningless, so I start the sum from $N=n,n+1,...

$f(n|p,\lambda) = \sum_{N=n}^{\infty} \binom{N}{n}p^{n}(1-p)^{N-n}\frac{\lambda^{N}e^{-\lambda}}{N!}$

Because, I do not know how to calculate it analytically, I calculate it numerically by generating a sequence of numbers $n,n+1,..., M$, where $M$ is a sufficiently large number that will make the $\textit{Poisson}$ prior zero (so I truncate the summation), and finally iteratively calculate the sum.

$2)$ Now that I have calculated (hope correctly) the prior predictive and I know all the terms of the posterior as functions of $N$, I can move forward to calculating numerically the $KL$ distance.

$$KL = \sum_{N=n}^{\infty} \pi(N|n,p)log\frac{\pi(N|n,p,\lambda)}{\pi(N|\lambda)}$$

I again simulate a large sequence of numbers $n,n+1,..., M$ for a sufficiently large $M$.


$i)$ Is there a way to calculate analytically the prior predictive $f(n|p,\lambda)$? Or what I did is used in practice??

$ii)$ In the second step, there would be a problem when $N=M$ (or when $N$ starts getting unlikely values) because the $\textit{Poisson}$ prior would be zero, something that has to be avoided because it is on the denominator. Is there a way to alleviate that problem?

$iii)$ The ranges we conduct the summations are correct?? I thought that they should be different because in the prior predictive we take the sum over the range of prior beliefs (however, we take into account the value of $n$ which is somehow counterintuitive). Whereas in the $KL$ the summation should be over the values of the posterior range, I assume they coincide?

Any suggestions would be great!


1 Answer 1


To start, yes, you're right, the Binomial distribution doesn't make sense for $N < n$. More precisely, we can say that $f(n | N, p) = 0$ for $N < n$ (intuitively, you're saying that the probability of getting $n$ successes in $N$ tries is zero). You can then write your summation as you did, starting from $N = n$. Then, you can analytically compute the prior predictive distribution as follows:

$p(n | p, \lambda) = \sum_{N=n}^{\infty} f(n | N, p) \pi(N|\lambda) = \sum_{N=n}^{\infty} {N \choose n} p^n (1-p)^{N-n} \frac{\lambda^N}{N!} e^{-\lambda}$

$= \frac{(\lambda p)^n}{e^\lambda} \sum_{N=n}^\infty \frac{N!}{n! (N-n)!} \frac{(\lambda(1-p))^{N-n}}{N!} = \frac{(\lambda p)^n}{n!e^\lambda} \sum_{N=n}^\infty \frac{(\lambda(1-p))^{N-n}}{(N-n)!}$

$= \frac{(\lambda p)^n}{n!e^\lambda} \sum_{k=0}^\infty \frac{(\lambda(1-p))^{k}}{k!} = \frac{(\lambda p)^n}{n!e^\lambda} e^{\lambda(1-p)} = \frac{(\lambda p)^n}{n!} e^{-\lambda p}$

You can see that this is again a Poisson distribution, with parameter $\lambda p$. Note that this is similar to a conjugate prior, but a rare one, because usually in a Binomial likelihood $N$ is considered known and $p$ is the target of statistical inference.

I hope this helps answer your questions!

  • 1
    $\begingroup$ Never thought about that, really helpful!!! $\endgroup$
    – Fiodor1234
    Jul 23, 2021 at 13:22
  • $\begingroup$ Do you think that it would be possible to use the same argument when you have more than one binomial draw, i.e. when your likelihood is of the form $\prod_{i=1}^{M}\binom{N}{n_{i}}p^{n_{i}}(1-p)^{N-n_{i}}$? $\endgroup$
    – Fiodor1234
    Jul 23, 2021 at 13:50
  • $\begingroup$ Honestly, I'm not sure. Your likelihood could become much more complex depending on the exact data generating process you have in mind. Could you describe that first, so we can see where that leads us? $\endgroup$
    – Maurits M
    Jul 23, 2021 at 16:57
  • $\begingroup$ Yes of course! There are $K$ occasions and for each one we sample $n_{k}$ individuals from a fixed population $N$. The occasions are independent. So, we can describe the observed data with the following likelihood $p(n_{1},n_{2},...n_{K}|N,p) = \prod_{k=1}^{K} Bin(n_{k};N,p)$. $\endgroup$
    – Fiodor1234
    Jul 23, 2021 at 19:07
  • $\begingroup$ Where because we work in a Bayesian framework we place a Poisson prior on $N$, you can regard the $p$ as know number because is out of interest $\endgroup$
    – Fiodor1234
    Jul 23, 2021 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.