6
$\begingroup$

Assuming I need to find the ML estimator for p, p being the chance of success in a Binomial experiment $Bin(N,p)$, I would expect my density function to be:

$$ f(y) = {{N}\choose{y}} p^y(1-p)^{N-y} $$

And so my likelihood function should be:

$$L(p) = \prod_i^n(f(y_i)) = \prod_i^n({{N}\choose{y_i}})p^{\sum_i^n y_i} (1-p)^{nN-\sum_i^n yi}$$

However, the last exponent, $nN-\sum yi$ seems to be wrong because when I derive the log likelihood and isolate $p$, I get $p = \frac{\sum_i^ny_i}{nN}$, while it should be $p = \frac{\sum_i^ny_i}{N}$. Indeed, when checking online, I find different exponents:

Here they have $N-\sum yi$, for no obvious reason to me (last time I checked, $\sum_i^n(N-y_i) = nN-\sum y_i$ !)

Here as well, but there they start from the likelihood function of a Bernoulli experiment. It makes sense that $\sum_i^n(1-y_i) = n-\sum y_i$, but what is more obscure to me is why they take the likelihood function of a Bernoulli while the problem is clearly about a Binomial. I am aware of the link between the two, but not enough to see why their likelihood functions seem to be substitutable to estimate p, especially since it doesn't give me the same result.

$\endgroup$
  • 1
    $\begingroup$ The estimator you suggest is a priori in (0,n), which is odd for an estimator of the parameter p which is in (0,1). This shows how to correct your approach. $\endgroup$ – Did May 1 '15 at 12:52
  • $\begingroup$ Why do you want $\sum_i y_i/N$ as an estimator of $p$? If this is for homework or suchwise, please add the self-study tag and read the corresponding wiki. $\endgroup$ – Xi'an May 1 '15 at 13:36
  • $\begingroup$ That is ok. Because the probability, $p$, is the same for both Bernoulli and Binomial experiments, they just prefered the simpler case. (note that in Bernoulli case $y \in \{0,1\}$ but in binomial $y \in \{0,\ldots,N\}$) $\endgroup$ – TPArrow May 1 '15 at 13:38
  • $\begingroup$ @Hamed: could you make your comment clearer? I do not understand how it relates to the OP question. $\endgroup$ – Xi'an May 1 '15 at 13:49
6
$\begingroup$

If you have a bernoulli experiment and repeat that (independently) N times, then you get a binomial variable. Then if you repeat a binomial experiment $n$ times that means you have repeated $nN$ bernoulli experiments.

Lets give you an example:

Assume $Y\sim Bin(p=3/4,N=5)$ and your observations after $n=5$ repetition are $ 5, 4, 2, 3, 4$ . Then it is clear that for example (5+ 4+ 2+ 3,+ 4)/5=3.6 is not an estimator of $p$ but $(5+4+2+3+4)/(5*5)=.72$ is.

$\endgroup$
3
$\begingroup$

to get MLE, you repeat Binomial Experiment with N trials n times.

So that,

first N trials give you $y_1$ success.

second N trials give you $y_2$ success.

.

.

.

nth N trials give you $y_n$ success.

Mathematically, you get MLE $\hat p=\frac{\sum\limits_{i=1}^ny_i}{nN}$(that is nothing but $\frac{total~success}{total~trials}$)

$\hat p=\frac{\sum\limits_{i=1}^ny_i}{N}$ is neither Mathematically correct nor logically(it gives you MLE for Expected success).

$\endgroup$
2
$\begingroup$

Assuming that you are talking about $n$ iid trials of $X_i \sim \operatorname{Binom}(N,p)$, the likelihood function you calculated is certainly correct:

$$L(p) = \prod_i^n(f(y_i)) = \prod_i^n \left[ {{N}\choose{y_i}}p^{y_i} (1-p)^{N- yi} \right] = \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}} \,.$$

By definition then we have that the MLE for $p$ is: $$\hat{p} = \arg\max_p \left[ \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}}\right] $$ Since $x \mapsto \ln x$ is a strictly increasing function of $x$, we have that $x_1 < x_2 \iff \ln x_1 < \ln x_2$ for all $x_1, x_2$ in the domain of this function, which includes the values of the likelihood we calculated above. Taking this fact into consideration, we get that:

$$\begin{array}{rcl} \hat{p} & = & \displaystyle\arg\max_p\ \ \ln\left[ \left[ \prod_i^n {{N}\choose{y_i}} \right] p^{\sum_1^n y_i} (1-p)^{nN - \sum_1^n{y_i}}\right] \\ & = & \displaystyle \arg\limits\max_p \sum_{i=1}^n\left[ \ln {{N}\choose{y_i}} + \left( \sum_{i=1}^n y_i \right)\ln(p) + \left( nN - \sum_{i=1}^n y_i \right)\ln(1-p) \right] \end{array}$$

Since this is a differentiable function of $p$, if a MLE $\hat{p}$ exists, by the first derivative test for local extrema, we will have that $\frac{\partial}{\partial p}$ of the above expression is equal to $0$ when $p = \hat{p}$ (provided $p$ is in the interior of $[0,1]$, and not one of the endpoints $p=0$ or $p=1$, which is pretty obvious since in both of those cases $L(p) = 0$). We are even guaranteed that this value of $p$ will not only be an extremum, but even a maximum, if we can show that the second derivative ($w.r.t. p$) of the above function is strictly less than zero, i.e. that the above function is concave.

(I claim, but do not show that this log-likelihood is concave. You can show this either by direct verification, e.g. page 4 here, or by showing that the binomial is an exponential family, so that in its natural/canonical parametrization its log-likelihood is concave (see here or here), therefore the MLE for the natural parameter is unique, then if you can show that $p$ is a one-to-one increasing function $h$ of the natural parameter $\eta:= \ln(p/1-p)$, you are done, i.e. just take $\hat{p} = h(\hat{\eta})$.)

TL;DR The value of $p$ such that the derivative of the above expression w.r.t. $p$ evaluates to $0$ is our MLE $\hat{p}$, i.e. we choose $\hat{p}$ so that:

$$0 + \frac{\left( \sum_{i=1}^n y_i \right)}{\hat{p}} - \frac{\left(n N - \sum_{i=1}^n y_i \right)}{1-\hat{p}} = 0 \,.$$

As one of the answers above claims, this means after doing some algebra that $$\hat{p} = \frac{\sum_{i=1}^n y_i}{nN} = \frac{\bar{y}}{N} \,. $$

$\endgroup$
1
$\begingroup$

In a Binomial experiment, we are interested in the number of successes: not a single sequence.

When calculating the Likelihood function of a Binomial experiment, you can begin from 1) Bernoulli distribution (i.e. single trial) or 2) just use Binomial distribution (number of successes)

1) Likelihood derived from Bernoulli trial

The probability of success of a single trial is \begin{align} P(y \mid p) = p^y(1-p)^{1-y} \end{align} and for a sequence of trials \begin{align} P(y_1,...,y_N \mid p) &= \prod_{i=1}^Np^{y_i}(1-p)^{1-y_i} \\ &=p^{\sum_{i=1}^Ny_i}(1-p)^{N-\sum_{i=1}^Ny_i}. \end{align}

For clarity, let's define the number of successes as \begin{align} k = \sum_{i=1}^Ny_i \end{align}

Giving us: \begin{align} P(y_1,...,y_N \mid p) &= p^{k}(1-p)^{N-k}. \end{align}

However, we are not interested in this single sequence, but all the sequences that produce similar number of successes.

This is similar to the relationship between the Bernoulli trial and a Binomial distribution: The probability of sequences that produce $k$ successes is given by multiplying the probability of a single sequence above with the binomial coefficient $\binom{N}{k}$.

Thus the likelihood (probability of our data given parameter value): \begin{align} L(p) = P(Y \mid p) &= \binom{N}{k}p^{k}(1-p)^{N-k}. \end{align}

2) Likelihood derived from Binomial distribution

The Binomial probability \begin{align} P(Y \mid p) &= \binom{N}{k}p^{k}(1-p)^{N-k}. \end{align}

already is the probability of $k$ successes over $N$ trials, not a single observation or a single sequence of observations.

Thus the Likelihood is not a product of these -- this would be the likelihood of several (independent) binomial experiments repeated, which is what you were getting at in your question!


Where the confusion comes from?

A lot of sources simply drop the binomial coefficient of the Likelihood function

\begin{align} L(p) \propto p^{k}(1-p)^{N-k}, \end{align}

without actually stating that this being done, or are simply not rigorous enough in their derivation: Using the likelihood of a single sequence instead.

Given fixed observations, $\binom{N}{k}$ is a constant and thus doesn't affect calculating MLE estimate or MCMC sampling from the posterior, and this is why they can get away with the mistake.

$\endgroup$

protected by kjetil b halvorsen Jun 26 '18 at 23:35

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.