I am looking for the maximum likelihood estimator (MLE) for the Poisson-binomial distribution. I understand the derivation of the MLE for a Poisson distribution and a binomial distribution, but I am unable to derive the MLE equation for the Poisson-binomial distribution.


1 Answer 1


Consider a Poisson-binomial distribution with probability parameter $\boldsymbol{\theta} \equiv (\theta_1, ..., \theta_m) \in [0,1]^m$. The probability density function (PDF) for this distribution is given by:

$$p(k | \boldsymbol{\theta}) = \sum_{\mathcal{A} \in \mathfrak{P}_k(m)} \Bigg( \prod_{j \in \mathcal{A}} \theta_j \Bigg) \Bigg( \prod_{j \notin \mathcal{A}} (1-\theta_j) \Bigg),$$

where $\mathfrak{P}_k(m)$ is the class of all subsets of $k$ labels from $\{ 1, 2, ..., m \}$. The sampling density is clearly invariant to permutations of the probability vector $\boldsymbol{\theta}$, and so this parameter is unidentifiable in the distribution. The minimal sufficient parameter for this distribution is the vector of ordered probability values $\boldsymbol{\theta}_*= (\theta_{(1)} \geqslant ... \geqslant \theta_{(m)})$. This means that the MLE for $\boldsymbol{\theta}$ cannot be estimated uniquely, and for any MLE vector, any permutation of that vector will also be an MLE.

Setting aside this identifiability issue, we can obtain equations for the MLE. Letting $\boldsymbol{\theta}_{-a} \in [0,1]^{m-1}$ be the probability parameter excluding the element $\theta_a$, we have the recursive formula:

$$p(k | \boldsymbol{\theta}) = \theta_a p(k-1 | \boldsymbol{\theta}_{-a}) + (1-\theta_a) p(k | \boldsymbol{\theta}_{-a}).$$

We therefore have the useful preliminary result:

$$\frac{\partial p}{\partial \theta_a} (k | \boldsymbol{\theta}) = p(k-1 | \boldsymbol{\theta}_{-a}) - p(k | \boldsymbol{\theta}_{-a}).$$

Now, given the observed vector $\boldsymbol{k} \equiv (k_1, ..., k_n)$ taken from IID draws from the Poisson-binomial distribution, we have log-likelihood function $l_{\boldsymbol{k}} (\boldsymbol{\theta}) = \sum_{i=1}^n \ln p(k_i | \boldsymbol{\theta})$ which gives us the score:

$$\frac{\partial l_\boldsymbol{k}}{\partial \theta_a} (\boldsymbol{\theta}) = \sum_{i=1}^n \frac{\partial}{\partial \theta_a} \ln p(k_i | \boldsymbol{\theta}) = \sum_{i=1}^n \frac{p(k_i-1 | \boldsymbol{\theta}_{-a}) - p(k | \boldsymbol{\theta}_{-a})}{p(k_i | \boldsymbol{\theta})}.$$

The MLE occurs at any point $\hat{\boldsymbol{\theta}}$ satisfying:

$$\sum_{i=1}^n \frac{p(k_i-1 | \hat{\boldsymbol{\theta}}_{-a}) - p(k | \hat{\boldsymbol{\theta}}_{-a})}{p(k_i | \hat{\boldsymbol{\theta}})} = 0 \text{ } \text{ } \text{ } \text{ for all } a=1,2,...,m.$$

We have already noted that the MLE is invariant to permutations, and therefore we only require a "representative" of the class of permutations. We can obtain this by imposing the additional order constraint $\theta_{1} \geqslant ... \geqslant \theta_{m}$. For $n \geqslant m$ this set of $m$ score equations --plus the ordering constraint-- should yield a unique solution for $\hat{\boldsymbol{\theta}}$. This solution represents the class of $m!$ permutations of this vector that are all MLEs.

  • $\begingroup$ How can $n>=m$ ?, I didn't understand the use of $p(k|\hat{\theta}_{-a})$ in MLE equation as it will be always 0. Can you elaborate it?. So n<=m ? $\endgroup$ Commented Mar 30, 2018 at 14:47
  • 1
    $\begingroup$ When you are looking for an MLE for a distribution, you are generally dealing with a situation where you observe multiple IID draws from that distribution. In this exposition of the problem, each draw from the distribution has $m$ trials, and you are taking $n$ IID draws from the distribution. There is no restriction on $m$ and $n$ in this case; the number of draws from the distribution can exceed the number of trials in any given draw. $\endgroup$
    – Ben
    Commented Mar 30, 2018 at 22:01
  • $\begingroup$ Maximisation of the likelihood is done using the usual calculus technique of finding the critical points. These occur at the points where the derivative of the log-likelihood is zero (the score equation). Because the parameter is a vector with length $m$, the score equation is a set of $m$ equations. In this particular case, the derivation of these derivatives show that the score equation involves terms of the form $p( k | \hat{\boldsymbol{\theta}}_{-a} )$. Those individual terms are not zero; they are probabilities of draws from a smaller Poisson-Binomial distribution. $\endgroup$
    – Ben
    Commented Mar 30, 2018 at 22:05
  • $\begingroup$ So in the above equation MLE occur at $\hat\theta$, then what is exact meaning of $\hat\theta_{-a}$ ?? Can you explain with simple example? $\endgroup$ Commented Apr 1, 2018 at 12:11
  • 1
    $\begingroup$ @jtlz2: Yes, that is a consequence of the non-identifiability of the parameter vector. In fact, not only is the likelihood multimodal, it has total symmetry with respect to permutations of $\boldsymbol{\theta}$. $\endgroup$
    – Ben
    Commented Apr 24, 2023 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.