Prolonged negative binomial beyond Poisson? The negative binomial distribution can parameterized with $\mu$ (mean) and $\sigma$ (standard deviation) with $\text{NB}(\mu,\sigma)$. While this parameterization is a bit unusual, it sheds light on $ \lim_{\sigma \to \sqrt \mu} \text{NB}(\mu, \sigma) = \text{Poisson}(\mu)$.
Is there a way to generalize the notion of negative binomial to go beyond the $\sqrt \mu$ limit to the standard deviation? While the negative binomial can be interpreted as a over dispersed Poisson, the generalization would be akin to an under dispersed Poisson when $\sigma < \sqrt \mu$.
Assuming that $\mu$ is an integer, it would be reasonable to expect the generalization to converge to a Dirac on $\mu$ when the standard deviation converges to zero. It would also be reasonable to expect the generalization to exhibit continuous variations of probabilities on both sides around $\sqrt \mu$ .
 A: The answer is yes: the prolongating distribution is the Binomial distribution.  The trilogy: Binomial|Poisson|Negative Binomial can be regarded as one
single two-parameter distribution for a non-negative integer r.v. $N$,
each probability $\text{Pr}\{N = n\}$ for $n \geqslant 0$ being a
smooth function of the parameter vector. I will first recall some
facts about a well-known trilogy forming a single distribution.
The Generalised Pareto Distribution (GPD)
Recall that the two-parameter GPD for a r.v. $X \geqslant 0$ involves
a scale parameter $\sigma_X >0$ and a shape parameter $\xi_X$. The
survival $S_X(x) := \text{Pr}\{X > x \}$ given by
$$ \tag{1} S_{X}(x) =
\begin{cases}
   \left[ 1 + \xi_X \,
   x /\sigma_X \right]_{+}^{-1/\xi}
   & \text{ if }\xi_X \neq 0, \\
   \exp\{ - x / \sigma_X \}
   & \text{ if }\xi_X = 0,
\end{cases}
   \qquad x \geqslant 0,
$$
where $z_+ := \max\{0,\, z\}$ for a real number $z$.
The GPD represents a trilogy of distributions corresponding to the
possible signs of the shape parameter $\xi_X$. For $\xi_X <0$ we get a
distribution with a finite upper end-point $-\sigma_X / \xi_X$, and
with no attractive name. The cases $\xi_X = 0$ and $\xi_X >0$
correspond to the famous exponential distribution and -up to a
re-parameterisation- to the Lomax
distribution. A
striking point is that the likelihood depends smoothly on the
parameter vector which is consistently used for the three
distributions. However there is no harm at considering the three
distributions for themselves. The three distributions correspond to
three ranges for the coefficient of variation $\text{CV}$: when $\xi_X
< 0$ we get $\text{CV} < 1$ or underdispersion, when $\xi_X >0$ we
get $\text{CV} > 1$ overdispersion, while $\text{CV} = 1$ in the
exponential case $\xi_X = 0$.
Another trilogy: Binomial|Poisson|Negative Binomial
Back to our three famous distributions: Binomial,
Poisson and Negative Binomial. I will not write what $\Pr\{N =
n\}$ is in each of the three cases, but instead recall that the
parameters are as follows

*

*Binomial:  size $\nu$ and probability $p$, with expectation
$\lambda := \nu p$.


*Poisson: rate $\lambda$, which is also the expectation.


*Negative binomial: size $\nu$ and probability $p$, with expectation
$\lambda := \nu p /q$ where $q := 1 - p$.
I retain here the parameterisation of the negative
binomial distribution of the stats  R package (?NegBinomial).
Remind that a random variable $N$ with integer value has no physical
dimension. For such a r.v. we can usefully consider the index of
dispersion
$\text{ID}$: the ratio variance / mean, which is dimensionless. This
leads to the terminology of under/over-dispersion for integer-valued
r.vs, which must not be confused with that for 'ordinary' non-negative
variables having a dimension which was used in the former section.
Binomial, Poisson and negative binomial correspond to $\text{ID} < 1$
(underdispersion), $\text{ID} = 1$ and $\text{ID}>1$ (overdispersion).
This can be viewed as an analogy with the GPD trilogy.
Now let us show that these three discrete distributions can be
regarded as one. To see this, consider the probability generating
function (p.g.f.)  $G_N(z) := \mathbb{E}[z^N]$ which is given by
$$
  \tag{2} 
   G_N(z) =
     \begin{cases}
    [1 - (1- z) \, p]^\nu & \text{binomial}, \\
    \exp\{-(1 - z) \,\lambda \}  & \text{Poisson}, \\
    \left[1 + (1 - z) \,p /q\right]^{-\nu} & \text{negative binomial},
     \end{cases}   
$$
which holds at least for for $z$ complex with $|z| < 1$.
Keeping in mind the expression for the expectation $\lambda$
corresponding to the three cases, it transpires that $G_N(z)$ relates to the
GPD survival $S_X(x)$ defined above through
$$
   \tag{3}
   G_N(z) =  S_{X}(1-z), \qquad \text{for }z \text{ real } 0 < z < 1,
$$
provided that the GPD scale is taken as $\sigma_N := 1/\lambda$ and
that the shape $\xi_N$ is given by
$$
   \xi_N := \begin{cases}
          -1/\nu & \text{binomial}, \\
      0  & \text{Poisson}, \\
          1 / \nu& \text{negative binomial}.
 \end{cases}   
$$
Now we can try to define a probability distribution for $N$ with two
parameters $\sigma_N >0$ and $\xi_N$ by using the formula
$$
    G_N(z) = 
    \left[ 1 + \xi_N \,\dfrac{1 - z}{\sigma_N}\right]^{-1/\xi_N}
    \qquad \text{if }
    \xi_N \neq 0.
$$
For that aim, we will impose the condition: $\sigma_N + \xi_N >0$. In
the binomial case when $\xi_N < 0$, this imposes that $p< 1$. This
condition tells as well that $x = 1$ is an interior point of the
support of the GPD with parameters $\sigma_N$ and $\xi_N$, and it
allows using the principal determination of the logarithm to correctly
define $G_N(z)$. While a non-integer value of $\nu > 0$ makes sense in
the negative binomial case, a non-integer $\nu$ is not possible in the
binomial case because the coefficients of the series expansion of
$G_N(z)$ would then fail to be non-negative.  So the parameter
"domain" $\Theta_N$ is formed by the couples $[\sigma_N, \, \xi_N]$
with $\sigma_N >0$ and $\xi_N \geq 0$ or $\xi_N$ being the inverse of
a negative integer with then $\sigma_N + \xi_N > 0$ (see Figure, left
panel). This is not an open set, but note that every point with
$\xi_N = 0$ is a cluster point.
Provided that $[\sigma_N,\,\xi_N]$ is in $\Theta_N$, we claim that
$G_N(z)$ is a p.g.f. This is quite obvious because we saw that for
each of the three cases $\xi_N >0$, $\xi_N=0$ and $\xi_N >0$ we get
the p.g.f. of a distribution of our trilogy as in (2). Yet the positivity of
the coefficients of the power series at $z=0$ could have been obtained
for $\xi_N >0$ as a consequence of the fact that the GPD survival is a
completely monotone function. For each possible value $n \geq 0$ of
$N$, the value of the density $p_N(n; \sigma_N,\,\xi_N) :=
\text{Pr}\{N = n\}$ if infinitely diffferentiable w.r.t. $[\sigma_N,\,
\xi_N]$ (see Figure, right panel) so it makes sense to consider
$G_N(z)$ as the p.g.f. of one single distribution which can be used
for ML estimation. Why not call this distribution Generalised
Binomial?
Alternative parameterisation
Instead of the two parameters $\sigma_N$ and $\xi_N$, we can use the
mean and the index of dispersion
$$ \mathbb{E}[N] = 1/\sigma_N, \quad \text{ID}(N) = 1 + \xi_N /
\sigma_N, $$ which leads to the inverse formula $$ \sigma_N = 1 /
\mathbb{E}[N], \quad \xi_N = \left\{\text{ID}(N) - 1 \right\} /
\mathbb{E}[N].  $$
The constraint $\sigma_N + \xi_N >0$ tells that $\text{ID}$ is positive.
For any given value $\nu$ of $\mathbb{E}(N)$ can have $\text{ID}
\approx 0$: this corresponds to the binomial distribution with
probability $p \approx 1$ i.e. to a Dirac distribution with it
mass at $\nu$, which hence must be an integer.
Remark: Maximum-Likelihood
Interestingly, if a sample $[X_i]$ of the GPD is available, the sign of
the ML estimate $\widehat{\xi}_X$ of the shape parameter depends in a
very simple way on the sample Coefficient of Variation
$\widehat{\text{CV}} := \{M_2/M_1^2 -1\}^{1/2}$, where $M_r$ is the
non-central sample moment of order $r$.  Indeed, it can be shown that
$\widehat{\xi}_X>0$ corresponds to the overdispersed case
$\widehat{\text{CV}} > 1$, while $\widehat{\xi}_X<0$ corresponds to
the underdispersed case $\widehat{\text{CV}} < 1$.  In the case where
$\widehat{\text{CV}}$ would exactly be equal to $1$, we would get the
exponential distribution $\widehat{\xi}_X = 0$. If we consider each of
the three distributions for itself, we may regard the ML estimation as
impossible: for instance the estimation for the Lomax distribution
when $\widehat{\text{CV}} < 1$.
Now consider the Generalised Binomial with an unknown size parameter -
although this is quite uncommon in the binomial setting. The
possibility of the ML estimation using a sample $[N_i]$ depends on the
sample index of dispersion $\widehat{\text{ID}} := M_2/ M_1$. It is
given by the conditions: $\widehat{\text{ID}} < 1$ for the binomial
case - see Blumenthal S. and Bahiya
R.C.,
and by $\widehat{\text{ID}} > 1$ in the negative binomial case. The
later statement has been known for some years as Anscombe's
conjecture for the Negative Binomial.

