1
$\begingroup$

I searched online, so many people mix up MLE of binomial and Bernoulli distribution.

They are saying: "If $X$ is $Binomial(N,\theta)$ then MLE is $\hat{\theta} = X/N$." And I don't agree with that, that is just MLE for Bernoulli in N trials, not for Binomial.

But I can't find correct Likelihood function and MLE of Binomial distribution online. So I try to derive it myself, and seek for confirmation here.


For Bernoulli, I know that::

its Likelihood function is $$\prod\limits_{i = 1}^n {{p_X}{{\left( {{x_i}} \right)}_{Ber\left( \theta \right)}}} = \left( {{\theta ^{\sum\limits_{i = 1}^n {{x_i}} }}{{\left( {1 - \theta } \right)}^{n - \sum\limits_{i = 1}^n {{x_i}} }}} \right)$$ its MLE is $${{\hat \theta }_{Ber\left( \theta \right)}} = \frac{{\sum\limits_{i = 1}^n {\left( {{x_i}} \right)} }}{n} = \bar x$$

${{p_X}\left( {{x_i}} \right)}$ is the pdf (or pmf)


For Binomial, I tried following::

its Likelihood function is

$$\left\{ \begin{array}{l}L\left( {\theta |{\bf{x}}} \right) = \prod\limits_{i = 1}^n {{p_X}{{\left( {{x_i}} \right)}_{Bin\left( {N,\theta } \right)}}} = \prod\limits_{i = 1}^n {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right) \cdot {\theta ^{{x_i}}}{{\left( {1 - \theta } \right)}^{N - {x_i}}}} \\ = \prod\limits_{i = 1}^n {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right)} \cdot \left( {\prod\limits_{i = 1}^n {{\theta ^{{x_i}}}{{\left( {1 - \theta } \right)}^{N - {x_i}}}} } \right)\\ = \prod\limits_{i = 1}^n {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right)} \cdot \left( {{\theta ^{\sum\limits_{i = 1}^n {{x_i}} }}{{\left( {1 - \theta } \right)}^{\sum\limits_{i = 1}^n {\left( {N - {x_i}} \right)} }}} \right)\\ = \prod\limits_{i = 1}^n {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right)} \cdot \left( {{\theta ^{\sum\limits_{i = 1}^n {{x_i}} }}{{\left( {1 - \theta } \right)}^{nN - \sum\limits_{i = 1}^n {{x_i}} }}} \right)\\\left[ {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right){\text{ is just a constant when }}{x_i}{\text{ is given}}} \right.\\ \propto {\theta ^{\sum\limits_{i = 1}^n {{x_i}} }}{\left( {1 - \theta } \right)^{nN - \sum\limits_{i = 1}^n {{x_i}} }}\end{array} \right.$$

(I don't think we can have a general formula for the constant, so I drop it and just use the proportion, if you know please tell me.)

its MLE is $$\left\{ \begin{array}{l}\ln L = \ln \left( {\prod\limits_{i = 1}^n {\left( {\begin{array}{*{20}{c}}N\\{{x_i}}\end{array}} \right)} } \right) + \sum\limits_{i = 1}^n {\left( {{x_i}} \right)} \cdot \ln \left( \theta \right) + \left( {nN - \sum\limits_{i = 1}^n {{x_i}} } \right) \cdot \ln \left( {1 - \theta } \right)\\\frac{{d\left( {\ln L} \right)}}{{d\theta }} = 0 + \frac{{\sum\limits_{i = 1}^n {\left( {{x_i}} \right)} }}{\theta } - \frac{{nN - \sum\limits_{i = 1}^n {{x_i}} }}{{1 - \theta }}\\\frac{{d\left( {\ln L} \right)}}{{d\hat \theta }} = \frac{{\sum\limits_{i = 1}^n {\left( {{x_i}} \right)} }}{{\hat \theta }} - \frac{{nN - \sum\limits_{i = 1}^n {{x_i}} }}{{1 - \hat \theta }} = 0\\\left( {1 - \hat \theta } \right) \cdot \sum\limits_{i = 1}^n {\left( {{x_i}} \right)} = \left( {nN - \sum\limits_{i = 1}^n {{x_i}} } \right) \cdot \hat \theta \\\sum\limits_{i = 1}^n {\left( {{x_i}} \right)} = \left( {nN - \sum\limits_{i = 1}^n {{x_i}} + \sum\limits_{i = 1}^n {{x_i}} } \right) \cdot \hat \theta \\{{\hat \theta }_{Bin\left( {N,\theta } \right)}} = \frac{{\sum\limits_{i = 1}^n {\left( {{x_i}} \right)} }}{{nN}} = \frac{{\bar x}}{N}\end{array} \right.$$

Am I correct?


Also, n is the number of samples (trials), N is the number of times you flip it in 1 trial. ie::

in Bernoulli, you flip the coin n trials, you flip it 1 time each trial.

in Binomial, you flip the coin n trials, you flip it N times each trial.

(I guess this is why so many people mix these two up when calculating the Likelihood function)

$\endgroup$
1

2 Answers 2

3
$\begingroup$

These two models are statistically equivalent: $$ X_1,\dots,X_n \sim \text{Ber}(\theta), \quad \text{i.i.d.} $$ and $$ T \sim \text{Bin}(n, \theta). $$ the latter being the reduction of the former by sufficiency.

If you consider the following problem: $$ Y_1,\dots, Y_n \sim \text{Bin}(N,\theta), \quad \text{i.i.d.} $$ This is a different problem than either of the two above, a different model, not equivalent to the previous ones statistically. (This third model is equivalent to a version of the two above with $n$ replaced with $nN$.)

Saying "people mix up MLE of binomial and Bernoulli distribution." is itself a mix-up. There is no MLE of binomial distribution. Similarly, there is no MLE of a Bernoulli distribution. You have to specify a "model" first. Then, you can ask about the MLE. There many different models involving Bernoulli distributions. There are also many different models involving Binomial distributions. Once you fix a model, you can talk about the likelihood, etc.

$\endgroup$
2
  • $\begingroup$ I do think the model is default to be a set of n samples with a distribution (with parameters), instead of 1 sample, when we are talking about Likelihood functions. ie: $X \sim {\text{Distribution}}\left( \theta \right)\& \left\{ {{x_1}, \cdots ,{x_n}} \right\}{\text{ are }}n{\text{ samples from }}X$ Shouldn't it? $\endgroup$
    – Nor.Z
    Jan 17, 2021 at 22:29
  • $\begingroup$ It all depends on you and the problem you want to set up. The model can be whatever you want. A single Bernoulli observation $X \sim \text{Ber}(\theta)$ is a perfectly fine model. $\endgroup$
    – passerby51
    Jan 17, 2021 at 23:44
0
$\begingroup$

Graphical comment: Suppose you have $X\sim\mathsf{Binom}(N, p),$ with $N=10$ and $x = 6$ successes.

p=seq(0,1, by=.01)
like = p^6*(1-p)^4
mle = p[like==max(like)];  mle
[1] 0.6

plot(p, like, type="l")
  abline(h=0, col="green2")
  abline(v = mle, col="red", lwd=2)

enter image description here

Suppose you have $X\sim\mathsf{Binom}(N, p),$ with $N=100$ and $x = 60$ successes.

p=seq(0,1, by=.01)
like = p^60*(1-p)^40
mle = p[like==max(like)];  mle
[1] 0.6
plot(p, like, type="l")
  abline(h=0, col="green2")
  abline(v = mle, col="red", lwd=2)

enter image description here

Notice that the curvature of the likelihood function at its maximum is much tighter than for $N = 10.$

$\endgroup$
3
  • $\begingroup$ My point is, if you are calculating the MLE of Binomial distribution, in general, you should use eg:$\left\{ {{x_1} = 6,{x_2} = 7,{x_3} = 7, \cdots ,{x_n} = 5} \right\}$ as your samples each with 10 flips, instead of only using 1 sample {x=6} with 10 flips. Or, I have to view it as 10 samples for a Bernoulli distribution instead of a Binomial distribution. Because only using 1 sample for calculating a MLE of a distribution is generally not good. Not sure if you get what I mean. $\endgroup$
    – Nor.Z
    Jan 17, 2021 at 21:03
  • $\begingroup$ I guess my point is that you're making it more complicated than it needs to be. Only the sufficient statistic matters. $\endgroup$
    – BruceET
    Jan 17, 2021 at 21:14
  • $\begingroup$ It might look complicated, but I think it add more clarification. I was trying to make a clarification that there is difference between MLE of Binomial and Bernoulli distribution. If people are referring to MLE of Bernoulli, then they should say its MLE of Bernoulli instead of saying that is MLE of Binomial (with n=1 sample). $\endgroup$
    – Nor.Z
    Jan 17, 2021 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.