31
$\begingroup$

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as

$L(p) = \prod_{i=1}^np^{x_i}(1-p)^{1-x_i}$

How to arrive at this equation? It seems pretty clear to me regarding the other distributions, Poisson and Gaussian;

$L(\theta) = \prod_{i=1}^n \text{PDF or PMF of dist.}$

But the one for binomial is just a little different. To be straight forward, how did

$nC_x~p^x(1-p)^{n-x}$

become

$p^{x_i}(1-p)^{1-x_i}$

in the above likelihood function?

$\endgroup$
1
34
$\begingroup$

In maximum likelihood estimation, you are trying to maximize $nC_x~p^x(1-p)^{n-x}$; however, maximizing this is equivalent to maximizing $p^x(1-p)^{n-x}$ for a fixed $x$.

Actually, the likelihood for the gaussian and poisson also do not involve their leading constants, so this case is just like those as w


Addressing OPs Comment

Here is a bit more detail:

First, $x$ is the total number of successes whereas $x_i$ is a single trial (0 or 1). Therefore:

$$\prod_{i=1}^np^{x_i}(1-p)^{1-x_i} = p^{\sum_1^n x_i}(1-p)^{\sum_1^n1-x_i} = p^{x}(1-p)^{n-x}$$

That shows how you get the factors in the likelihood (by running the above steps backwards).

Why does the constant go away? Informally, and what most people do (including me), is just notice that the leading constant does not affect the value of $p$ that maximizes the likelihood, so we just ignore it (effectively set it to 1).

We can derive this by taking the log of the likelihood function and finding where its derivative is zero:

$$\ln\left(nC_x~p^x(1-p)^{n-x}\right) = \ln(nC_x)+x\ln(p)+(n-x)\ln(1-p)$$

Take derivative wrt $p$ and set to $0$:

$$\frac{d}{dp}\ln(nC_x)+x\ln(p)+(n-x)\ln(1-p) = \frac{x}{p}- \frac{n-x}{1-p} = 0$$

$$\implies \frac{n}{x} = \frac{1}{p} \implies p = \frac{x}{n}$$

Notice that the leading constant dropped out of the calculation of the MLE.

More philosophically, a likelihood is only meaningful for inference up to a multiplying constant, such that if we have two likelihood functions $L_1,L_2$ and $L_1=kL_2$, then they are inferentially equivalent. This is called the Law of Likelihood. Therefore, if we are comparing different values of $p$ using the same likelihood function, the leading term becomes irrelevant.

At a practical level, inference using the likelihood function is actually based on the likelihood ratio, not the absolute value of the likelihood. This is due to the asymptotic theory of likelihood ratios (which are asymptotically chi-square -- subject to certain regularity conditions that are often appropriate). Likelihood ratio tests are favored due to the Neyman-Pearson Lemma. Therefore, when we attempt to test two simple hypotheses, we will take the ratio and the common leading factor will cancel.

NOTE: This will not happen if you were comparing two different models, say a binomial and a poisson. In that case, the constants are important.

Of the above reasons, the first (irrelevance to finding the maximizer of L) most directly answers your question.

$\endgroup$
2
  • 2
    $\begingroup$ We can see that's the idea. But could you explain a little more on how $nC_x$ is removed and $n$ is replaced with 1? $\endgroup$
    – Ébe Isaac
    Nov 11 '15 at 4:31
  • $\begingroup$ @ÉbeIsaac added some more details $\endgroup$
    – user75138
    Nov 11 '15 at 5:16
2
$\begingroup$

xi in the product refers to each individual trial. For each individual trial xi can be 0 or 1 and n is equal to 1 always. Therefore, trivially, the binomial coefficient will be equal to 1. Hence, in the product formula for likelihood, product of the binomial coefficients will be 1 and hence there is no nCx in the formula. Realised this while working it out step by step :) (Sorry about the formatting, not used to answering with mathematical expressions in answers...yet :) )

$\endgroup$
1
  • 1
    $\begingroup$ It's important to recognize that you are conceiving of the data in a particular way: namely, you are distinguishing the order in which the values were collected. But that's not an apparent part of the problem, which means the binomial factor really does belong in the likelihood. Thus, we need to appeal to some of the answers in this thread for the real reason why the binomial factor does not appear. $\endgroup$
    – whuber
    Jun 10 '20 at 18:45
0
$\begingroup$

It might help to remember that likelihoods are not probabilities. In other words, there is no need to have them sum to 1 over the sample space. Therefore, to make the math happen more quickly we can remove anything that is not a function of the data or the parameter(s) from the definition of the likelihood function.

$\endgroup$
0
$\begingroup$

For each factor in the likelihood (i.e. for each individual) "n" = $1$ and "x" = $0$ or $1$. In this case with ($n=1$) we always have $C_x = 1$. So $n C_x = 1$ for each of the factors making up the likelihood. So the normalization IS there, it is just $1$.

In general, a good check that one has written down the likelihood correctly and completely (i.e. including all of the factors, even if they do not affect an MLE calculation) is that if you sum the likelihood over all possible realizations of the data you get $1$. It is easy to see that the Miller and Freund's formula is normalized to $1$ this way (just sum over all $x_i = 0$ and $x_i =1$ for all of the i's, one gets $(1-p) + p = 1$ for each $i$ factor)

$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.