In the paper Small-sample estimation of negative binomial dispersion, with applications to SAGE data, section 4.1 Conditional maximum likelihood:

For RNA sequencing data, assume the counts for a single tag across $n$ libraries is a negative binomial random variable. Consider $Y_1, \cdots , Y_n$ as independent and $NB(\mu_i = m_i \lambda, \phi)$, where $m_i$ is the library size (i.e. total number of tags sequenced for library $i$) and $\lambda$ represents the proportion of the library that is a particular tag. The probability mass function is:

$$f(y_i;\mu,\phi)=P(Y=y_i)=\frac{\Gamma(\phi^{-1}+y_i)}{\Gamma(\phi^{-1})\Gamma(y_i+1)}\left(\frac{1}{\phi^{-1}\mu^{-1}+1}\right)^{y_i}\left(\frac{1}{\phi\mu+1}\right)^{\phi^{-1}} \tag 1$$

$$\text{E}(Y)=\mu$$ and $$\text{Var}(Y)=\mu+\phi\mu^2$$

If all libraries are the same size (i.e. $m_i ≡ m$), the sum $$Z = Y_1 + \cdots + Y_n \sim NB(nm\lambda, \phi n^{−1})$$

How to derive conditional maximum likelihood for $\phi$ that is independent of $\lambda$?


  • $\begingroup$ Given that there is a prespecified number of tags per library, the distribution can't be negative binomial, which has no upper bound. The problem description makes it look like a standard binomial distribution; why do you believe it's overdispersed relative to the binomial? $\endgroup$
    – jbowman
    Feb 20 at 2:18
  • $\begingroup$ I edited the question to add the original literature. The counts of a tag in different libraries following an NB distribution is a common treatment. $\endgroup$
    – Dan Li
    Feb 20 at 17:58
  • $\begingroup$ What do you mean by "independent of $\lambda$"? If $\lambda = 0.1$, the MLE for $\phi$ will be different than if $\lambda = 0.9$, so... $\endgroup$
    – jbowman
    Feb 20 at 19:05
  • $\begingroup$ I read the paper and I see what they are referring to - the profile log-likelihood, where $\lambda$'s maximum likelihood estimate replaces $\lambda$. I'm not used to seeing that referred to as a "conditional maximum likelihood", but then that's just me. $\endgroup$
    – jbowman
    Feb 20 at 20:13
  • 1
    $\begingroup$ @jbowman Conditional likelihod is not profile likelihood. $\endgroup$ Mar 4 at 23:02

1 Answer 1


Based on the Conditional Likelihood defined in https://rss.onlinelibrary.wiley.com/doi/epdf/10.1111/j.2517-6161.1996.tb02101.x, the conditional log-likelihood for $\phi$ of $Y$ conditioned on $Z$, dropping terms that don't involve $\phi$, is: $$l_{Y|Z=z}(\mathbf y; \phi)=\log f(\mathbf y;\mu,\phi)-\log f_{z}(z;n\mu,n^{-1}\phi)\\ =\sum\log \Gamma(y_i+\phi^{-1}) - n \log\Gamma(\phi^{-1}) + \sum y_i\log\left({\phi\mu \over 1 + \phi\mu}\right) + n\phi^{-1}\log\left({1 \over 1 + \phi\mu}\right)\\ -\log\Gamma(z+n\phi^{-1})+\log\Gamma(n\phi^{-1})-z\log\left({\phi\mu \over 1 + \phi\mu}\right) - n\phi^{-1}\log\left({1 \over 1 + \phi\mu}\right)\\ =\left[\sum_{i=1}^{n}\log\Gamma(y_i+\phi^{-1})\right]+\log\Gamma(n\phi^{-1})-\log\Gamma(z+n\phi^{-1})-n\log\Gamma(\phi^{-1})$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.