Maximum likelihood estimation of p in a Binomial sample

Question

Assuming I need to find the ML estimator for p, p being the chance of success in a Binomial experiment $Bin(N,p)$, I would expect my density function to be:

$$ f(y) = {{N}\choose{y}} p^y(1-p)^{N-y} $$

And so my likelihood function should be:

$$L(p) = \prod_i^n(f(y_i)) = \prod_i^n({{N}\choose{y_i}})p^{\sum_i^n y_i} (1-p)^{nN-\sum_i^n yi}$$

However, the last exponent, $nN-\sum yi$ seems to be wrong because when I derive the log likelihood and isolate $p$, I get $p = \frac{\sum_i^ny_i}{nN}$, while it should be $p = \frac{\sum_i^ny_i}{N}$. Indeed, when checking online, I find different exponents:

Here they have $N-\sum yi$, for no obvious reason to me (last time I checked, $\sum_i^n(N-y_i) = nN-\sum y_i$ !)

Here as well, but there they start from the likelihood function of a Bernoulli experiment. It makes sense that $\sum_i^n(1-y_i) = n-\sum y_i$, but what is more obscure to me is why they take the likelihood function of a Bernoulli while the problem is clearly about a Binomial. I am aware of the link between the two, but not enough to see why their likelihood functions seem to be substitutable to estimate p, especially since it doesn't give me the same result.

Answer 1

If you have a bernoulli experiment and repeat that (independently) N times, then you get a binomial variable. Then if you repeat a binomial experiment $n$ times that means you have repeated $nN$ bernoulli experiments.

Lets give you an example:

Assume $Y\sim Bin(p=3/4,N=5)$ and your observations after $n=5$ repetition are $ 5, 4, 2, 3, 4$ . Then it is clear that for example (5+ 4+ 2+ 3,+ 4)/5=3.6 is not an estimator of $p$ but $(5+4+2+3+4)/(5*5)=.72$ is.

Answer 2

Assuming that you are talking about $n$ iid trials of $X_i \sim \operatorname{Binom}(N,p)$, the likelihood function you calculated is certainly correct:

$$L(p) = \prod_i^n(f(y_i)) = \prod_i^n \left[ {{N}\choose{y_i}}p^{y_i} (1-p)^{N- yi} \right] = \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}} \,.$$

By definition then we have that the MLE for $p$ is: $$\hat{p} = \arg\max_p \left[ \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}}\right] $$ Since $x \mapsto \ln x$ is a strictly increasing function of $x$, we have that $x_1 < x_2 \iff \ln x_1 < \ln x_2$ for all $x_1, x_2$ in the domain of this function, which includes the values of the likelihood we calculated above. Taking this fact into consideration, we get that:

$$\begin{array}{rcl} \hat{p} & = & \displaystyle\arg\max_p\ \ \ln\left[ \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}}\right] \\ & = & \displaystyle \arg\limits\max_p \sum_{i=1}^n\left[ \ln {{N}\choose{y_i}} + \left( \sum_{i=1}^n y_i \right)\ln(p) + \left( nN - \sum_{i=1}^n y_i \right)\ln(1-p) \right] \end{array}$$

Since this is a differentiable function of $p$, if a MLE $\hat{p}$ exists, by the first derivative test for local extrema, we will have that $\frac{\partial}{\partial p}$ of the above expression is equal to $0$ when $p = \hat{p}$ (provided $p$ is in the interior of $[0,1]$, and not one of the endpoints $p=0$ or $p=1$, which is pretty obvious since in both of those cases $L(p) = 0$). We are even guaranteed that this value of $p$ will not only be an extremum, but even a maximum, if we can show that the second derivative ($w.r.t. p$) of the above function is strictly less than zero, i.e. that the above function is concave.

(I claim, but do not show that this log-likelihood is concave. You can show this either by direct verification, e.g. page 4 here, or by showing that the binomial is an exponential family, so that in its natural/canonical parametrization its log-likelihood is concave (see here or here), therefore the MLE for the natural parameter is unique, then if you can show that $p$ is a one-to-one increasing function $h$ of the natural parameter $\eta:= \ln(p/1-p)$, you are done, i.e. just take $\hat{p} = h(\hat{\eta})$.)

TL;DR The value of $p$ such that the derivative of the above expression w.r.t. $p$ evaluates to $0$ is our MLE $\hat{p}$, i.e. we choose $\hat{p}$ so that:

$$0 + \frac{\left( \sum_{i=1}^n y_i \right)}{\hat{p}} - \frac{\left(n N - \sum_{i=1}^n y_i \right)}{1-\hat{p}} = 0 \,.$$

As one of the answers above claims, this means after doing some algebra that $$\hat{p} = \frac{\sum_{i=1}^n y_i}{nN} = \frac{\bar{y}}{N} \,. $$

Answer 3

to get MLE, you repeat Binomial Experiment with N trials n times.

So that,

first N trials give you $y_1$ success.

second N trials give you $y_2$ success.

.

.

.

nth N trials give you $y_n$ success.

Mathematically, you get MLE $\hat p=\frac{\sum\limits_{i=1}^ny_i}{nN}$(that is nothing but $\frac{total~success}{total~trials}$)

$\hat p=\frac{\sum\limits_{i=1}^ny_i}{N}$ is neither Mathematically correct nor logically(it gives you MLE for Expected success).

Answer 4

In a Binomial experiment, we are interested in the number of successes: not a single sequence.

When calculating the Likelihood function of a Binomial experiment, you can begin from 1) Bernoulli distribution (i.e. single trial) or 2) just use Binomial distribution (number of successes)

1) Likelihood derived from Bernoulli trial

The probability of success of a single trial is \begin{align} P(y \mid p) = p^y(1-p)^{1-y} \end{align} and for a sequence of trials \begin{align} P(y_1,...,y_N \mid p) &= \prod_{i=1}^Np^{y_i}(1-p)^{1-y_i} \\ &=p^{\sum_{i=1}^Ny_i}(1-p)^{N-\sum_{i=1}^Ny_i}. \end{align}

For clarity, let's define the number of successes as \begin{align} k = \sum_{i=1}^Ny_i \end{align}

Giving us: \begin{align} P(y_1,...,y_N \mid p) &= p^{k}(1-p)^{N-k}. \end{align}

However, we are not interested in this single sequence, but all the sequences that produce similar number of successes.

This is similar to the relationship between the Bernoulli trial and a Binomial distribution: The probability of sequences that produce $k$ successes is given by multiplying the probability of a single sequence above with the binomial coefficient $\binom{N}{k}$.

Thus the likelihood (probability of our data given parameter value): \begin{align} L(p) = P(Y \mid p) &= \binom{N}{k}p^{k}(1-p)^{N-k}. \end{align}

2) Likelihood derived from Binomial distribution

The Binomial probability \begin{align} P(Y \mid p) &= \binom{N}{k}p^{k}(1-p)^{N-k}. \end{align}

already is the probability of $k$ successes over $N$ trials, not a single observation or a single sequence of observations.

Thus the Likelihood is not a product of these -- this would be the likelihood of several (independent) binomial experiments repeated, which is what you were getting at in your question!

---

Where the confusion comes from?

A lot of sources simply drop the binomial coefficient of the Likelihood function

\begin{align} L(p) \propto p^{k}(1-p)^{N-k}, \end{align}

without actually stating that this being done, or are simply not rigorous enough in their derivation: Using the likelihood of a single sequence instead.

Given fixed observations, $\binom{N}{k}$ is a constant and thus doesn't affect calculating MLE estimate or MCMC sampling from the posterior, and this is why they can get away with the mistake.