Maximum Likelihood Estimator for Negative Binomial Distribution

The question is the following:

A random sample of n values is collected from a negative binomial distribution with parameter k = 3.

1. Find the maximum likelihood estimator of the parameter π.
2. Find an asymptotic formula for the standard error of this estimator.
3. Explain why the negative binomial distribution will be approximately normal if the parameter k is large enough. What are the parameters of this normal approximation?

My working has been the following:
1. I feel like this is what is wanted but I'm not sure if I'm accurate here or if I can possibly take this further given the information provided? $$p(x) = {x-1 \choose k-1}\pi^k(1-\pi)^{x-k}\\ L(\pi) = \Pi_i^n p(x_n|\pi)\\ \ell(\pi) = \Sigma_i^n\ln(p(x_n|\pi))\\ \ell(\pi) = \Sigma_i^n\dfrac{k}{\pi}-\dfrac{(x-k)}{(1-\pi)}$$

1. I think the following is what is asked for. For the final part I feel like I need to replace $\hat{\pi}$ with $\dfrac{k}{x}$ $$\ell(\hat{\pi}) = -\dfrac{k}{\hat{\pi}^2} + \dfrac{x}{(1-\hat{\pi})^2}\\ se(\hat{\pi}) = \sqrt{-\dfrac{1}{\ell`(\hat{\pi})}}\\ se(\hat{\pi}) = \sqrt{\dfrac{\hat{\pi}^2}{k} - \dfrac{(1-\hat{\pi})^2}{x}}\\$$

2. I am not really sure how to prove this one and am still researching it. Any hints or useful links would be greatly appreciated. I feel like it is related either to the fact that a negative binomial distribution can be seen as a collection of geometric distributions or the inverse of a binomial distribution but not sure how to approach it.

Any help at all would be greatly appreciated

• (1) To find the maximum-likelihood estimate $\hat \pi$ you need to find where the log-likelihood function reaches its maximum. Calculating the score (the first derivative of the log-likelihood function with respect to $\pi$) is a start - what value will this take at the maximum? (And remember you don't need to estimate $k$.) – Scortchi - Reinstate Monica Aug 6 '15 at 12:17
• I forgot to add in the derivative of the log-likelihood = 0 for the purposes of figuring out the maximum. If I have figured this correctly (been working on it still since posting), what I have is $\dfrac{k}{\pi}-\dfrac{\Sigma_{i=0}^n(x_i-k)}{(1-\pi)} = 0$ – Syzorr Aug 6 '15 at 12:26
• Take care: $\sum_{i=1}^n \frac{k}{\pi} - \sum_{i=1}^n{\frac{(x_i-k)}{(1-\pi)}} = \ ?$ Also note that $i$ starts at 1. – Scortchi - Reinstate Monica Aug 6 '15 at 12:41
• In (2), it is rarely the case that the reciprocal of a difference is the difference of the reciprocals. This mistake hugely affects your final formula for $se(\hat\pi)$. – whuber Aug 6 '15 at 19:38

1.

$p(x) = {x_i-1 \choose k-1}\pi^k(1-\pi)^{x_i-k}$

$L(\pi;x_i) = \prod_{i=1}^{n}{x_i-1 \choose k-1}\pi^k(1-\pi)^{x_i-k}\\$

$\ell(\pi;x_i) = \sum_{i=1}^{n}[log{x_i-1 \choose k-1}+klog(\pi)+(x_i-k)log(1-\pi)]\\ \frac{d\ell(\pi;x_i)}{d\pi} = \sum_{i=1}^{n}[\dfrac{k}{\pi}-\dfrac{(x_i-k)}{(1-\pi)}]$

Set this to zero,

$\frac{nk}{\pi}=\frac{\sum_{i=1}^nx_i-nk}{1-\pi}$

$\therefore$ $\hat\pi=\frac{nk}{\sum_{i=1}^nx}$

2.

For second part you need to use the theorem that $\sqrt{n}(\hat\theta-\theta) \overset{D}{\rightarrow}N(0,\frac{1}{I(\theta)})$, $I(\theta)$ is the fisher information here. Therefore,the standard deviation of the $\hat\theta$ will be $[nI(\theta)]^{-1/2}$. Or you call it as standard error since you use CLT here.

So we need to calculate the Fisher information for the negative binomial distribution.

$\frac{\partial^2 \log(P(x;\pi))}{\partial\pi^2}=-\frac{k}{\pi^2}-\frac{x-k}{(1-\pi)^2}$

$I(\theta)=-E(-\frac{k}{\pi^2}-\frac{x-k}{(1-\pi)^2})=\frac{k}{\pi^2}+\frac{k(1-\pi)}{(1-\pi)^2\pi}$

Note: $E(x) =\frac{k}{\pi}$ for the negative binomial pmf

Therefore, the standard error for $\hat \pi$ is $[n(\frac{k}{\pi^2}+\frac{k(1-\pi)}{(1-\pi)^2\pi})]^{-1/2}$

Simplify we get we get $se(\pi)=\sqrt{\dfrac{\pi^2(\pi-1)}{kn}}$

3.

The geometric distribution is a special case of negative binomial distribution when k = 1. Note $\pi(1-\pi)^{x-1}$ is a geometric distribution

Therefore, negative binomial variable can be written as a sum of k independent, identically distributed (geometric) random variables.

So by CLT negative binomial distribution will be approximately normal if the parameter k is large enough

• Please read What topics can I ask about here? on self-study questions: rather than do people's homework for them we try to help them to do it themselves. – Scortchi - Reinstate Monica Aug 6 '15 at 14:33
• You do need to consider the sample size $n$ when calculating the MLE. You may be confusing an account of $n$ independent observations, each of the no. of trials required to reach $k$ failures ($x_1, x_2, \ldots, x_n$) with an account of a single observation of the no. of trials required to reach $k$ failures ($n$). The former gives a likelihood of $\sum_{i=1}^n{\pi^(1-\pi)^{x_i-k}}$; the latter, $\pi^k(1-\pi)^{n-k}$. – Scortchi - Reinstate Monica Aug 6 '15 at 14:34
• You are right, I am always confusing on this part. Thank you very much. I also ask a lot of questions on this board, but I really hope people can give me very detailed answer, then I can study it by myself step by step. – Deep North Aug 6 '15 at 22:24
• Yeah. I get why the rule against providing too much detail but this answer combined with my own notes from the lecture have allowed me to tie a lot of the loose ends together. I intend on going and speaking to my lecturer today about this so that I can get clarification from him. It is Friday here now. Assignment due Monday as stated above. We learned this on Wednesday and only have a single example using a binomial distribution. Thanks very much for the detail. – Syzorr Aug 6 '15 at 23:18
• There are some faults in your working there because I(θ) = E[] not -E[] (which has been confusing me until I went searching for the equations you've used) Eventually have ended up with $se(\pi)=\sqrt{\dfrac{\pi^2(\pi-1)}{kn}}$ – Syzorr Aug 8 '15 at 3:34