# How to calculate the conditional maximum likelihood of independent negative binomial variables conditioned on the likelihood of the sum?

In the paper Small-sample estimation of negative binomial dispersion, with applications to SAGE data, section 4.1 Conditional maximum likelihood:

For RNA sequencing data, assume the counts for a single tag across $$n$$ libraries is a negative binomial random variable. Consider $$Y_1, \cdots , Y_n$$ as independent and $$NB(\mu_i = m_i \lambda, \phi)$$, where $$m_i$$ is the library size (i.e. total number of tags sequenced for library $$i$$) and $$\lambda$$ represents the proportion of the library that is a particular tag. The probability mass function is:

$$f(y_i;\mu,\phi)=P(Y=y_i)=\frac{\Gamma(\phi^{-1}+y_i)}{\Gamma(\phi^{-1})\Gamma(y_i+1)}\left(\frac{1}{\phi^{-1}\mu^{-1}+1}\right)^{y_i}\left(\frac{1}{\phi\mu+1}\right)^{\phi^{-1}} \tag 1$$

$$\text{E}(Y)=\mu$$ and $$\text{Var}(Y)=\mu+\phi\mu^2$$

If all libraries are the same size (i.e. $$m_i ≡ m$$), the sum $$Z = Y_1 + \cdots + Y_n \sim NB(nm\lambda, \phi n^{−1})$$

How to derive conditional maximum likelihood for $$\phi$$ that is independent of $$\lambda$$?

$$l_{Y|Z=z}(\phi)=\left[\sum_{i=1}^{n}\log\Gamma(y_i+\phi^{-1})\right]+\log\Gamma(n\phi^{-1})-\log\Gamma(z+n\phi^{-1})-n\log\Gamma(\phi^{-1})$$

• Given that there is a prespecified number of tags per library, the distribution can't be negative binomial, which has no upper bound. The problem description makes it look like a standard binomial distribution; why do you believe it's overdispersed relative to the binomial? Feb 20 at 2:18
• I edited the question to add the original literature. The counts of a tag in different libraries following an NB distribution is a common treatment. Feb 20 at 17:58
• What do you mean by "independent of $\lambda$"? If $\lambda = 0.1$, the MLE for $\phi$ will be different than if $\lambda = 0.9$, so... Feb 20 at 19:05
• I read the paper and I see what they are referring to - the profile log-likelihood, where $\lambda$'s maximum likelihood estimate replaces $\lambda$. I'm not used to seeing that referred to as a "conditional maximum likelihood", but then that's just me. Feb 20 at 20:13
• @jbowman Conditional likelihod is not profile likelihood. Mar 4 at 23:02

Based on the Conditional Likelihood defined in https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.2517-6161.1996.tb02101.x, the conditional log-likelihood for $$\phi$$ of $$Y$$ conditioned on $$Z$$, dropping terms that don't involve $$\phi$$, is: $$l_{Y|Z=z}(\mathbf y; \phi)=\log f(\mathbf y;\mu,\phi)-\log f_{z}(z;n\mu,n^{-1}\phi)\\ =\sum\log \Gamma(y_i+\phi^{-1}) - n \log\Gamma(\phi^{-1}) + \sum y_i\log\left({\phi\mu \over 1 + \phi\mu}\right) + n\phi^{-1}\log\left({1 \over 1 + \phi\mu}\right)\\ -\log\Gamma(z+n\phi^{-1})+\log\Gamma(n\phi^{-1})-z\log\left({\phi\mu \over 1 + \phi\mu}\right) - n\phi^{-1}\log\left({1 \over 1 + \phi\mu}\right)\\ =\left[\sum_{i=1}^{n}\log\Gamma(y_i+\phi^{-1})\right]+\log\Gamma(n\phi^{-1})-\log\Gamma(z+n\phi^{-1})-n\log\Gamma(\phi^{-1})$$