How to derive the likelihood function for binomial distribution for parameter estimation?

Question

According to Miller and Freund's Probability and Statistics for Engineers, 8ed (pp.217-218), the likelihood function to be maximised for binomial distribution (Bernoulli trials) is given as

$L(p) = \prod_{i=1}^np^{x_i}(1-p)^{1-x_i}$

How to arrive at this equation? It seems pretty clear to me regarding the other distributions, Poisson and Gaussian;

$L(\theta) = \prod_{i=1}^n \text{PDF or PMF of dist.}$

But the one for binomial is just a little different. To be straight forward, how did

$nC_x~p^x(1-p)^{n-x}$

become

$p^{x_i}(1-p)^{1-x_i}$

in the above likelihood function?

Answer 1

In maximum likelihood estimation, you are trying to maximize $nC_x~p^x(1-p)^{n-x}$; however, maximizing this is equivalent to maximizing $p^x(1-p)^{n-x}$ for a fixed $x$.

Actually, the likelihood for the gaussian and poisson also do not involve their leading constants, so this case is just like those as w

---

Addressing OPs Comment

Here is a bit more detail:

First, $x$ is the total number of successes whereas $x_i$ is a single trial (0 or 1). Therefore:

$$\prod_{i=1}^np^{x_i}(1-p)^{1-x_i} = p^{\sum_1^n x_i}(1-p)^{\sum_1^n1-x_i} = p^{x}(1-p)^{n-x}$$

That shows how you get the factors in the likelihood (by running the above steps backwards).

Why does the constant go away? Informally, and what most people do (including me), is just notice that the leading constant does not affect the value of $p$ that maximizes the likelihood, so we just ignore it (effectively set it to 1).

We can derive this by taking the log of the likelihood function and finding where its derivative is zero:

$$\ln\left(nC_x~p^x(1-p)^{n-x}\right) = \ln(nC_x)+x\ln(p)+(n-x)\ln(1-p)$$

Take derivative wrt $p$ and set to $0$:

$$\frac{d}{dp}\ln(nC_x)+x\ln(p)+(n-x)\ln(1-p) = \frac{x}{p}- \frac{n-x}{1-p} = 0$$

$$\implies \frac{n}{x} = \frac{1}{p} \implies p = \frac{x}{n}$$

Notice that the leading constant dropped out of the calculation of the MLE.

More philosophically, a likelihood is only meaningful for inference up to a multiplying constant, such that if we have two likelihood functions $L_1,L_2$ and $L_1=kL_2$, then they are inferentially equivalent. This is called the Law of Likelihood. Therefore, if we are comparing different values of $p$ using the same likelihood function, the leading term becomes irrelevant.

At a practical level, inference using the likelihood function is actually based on the likelihood ratio, not the absolute value of the likelihood. This is due to the asymptotic theory of likelihood ratios (which are asymptotically chi-square -- subject to certain regularity conditions that are often appropriate). Likelihood ratio tests are favored due to the Neyman-Pearson Lemma. Therefore, when we attempt to test two simple hypotheses, we will take the ratio and the common leading factor will cancel.

NOTE: This will not happen if you were comparing two different models, say a binomial and a poisson. In that case, the constants are important.

Of the above reasons, the first (irrelevance to finding the maximizer of L) most directly answers your question.

Answer 2

It might help to remember that likelihoods are not probabilities. In other words, there is no need to have them sum to 1 over the sample space. Therefore, to make the math happen more quickly we can remove anything that is not a function of the data or the parameter(s) from the definition of the likelihood function.

Answer 3

$x_i$ in the product refers to each individual trial. For each individual trial $x_i$ can be $0$ or $1$ and $n$ is equal to $1$ always. Therefore, trivially, the binomial coefficient will be equal to $1$. Hence, in the product formula for likelihood, product of the binomial coefficients will be $1$ and hence there is no $nC_x$ in the formula.
Realised this while working it out step by step :)

Answer 4

I don't know why this old question has re-appeared in the current list, but I think that I can add something helpful that might be known by the previous respondents and perhaps implied rather obscurely in their answers. Likelihoods are only defined up to a proportionality constant. Fisher was quite explicit about that in the paper in which he first offered a definition of likelihood. Fisher (1922) On the mathematical foundations of statistics http://rsta.royalsocietypublishing.org/content/222/594-604/309

A consequence of being defined only to a proportionality constant is that you can use any proportionality constant that you like and the likelihood function will still be correct. The $nC_x$ is a proportionality constant and so it can be validly replaced with unity, as in the second version of the function in your post, or by 2 or 7.32 or any other constant.

That might seem to be silly and to make likelihoods unusable, but the main use of likelihoods is as a ratio and any proportionality constant will cancel out in such a ratio and have no influence on the result. The need to have the constants cancel does bring a requirement that the likelihoods in a ratio be scaled by the same proportionality constant. That is not really a problem because the likelihoods in a ratio will usually be from the same statistical model and so they will naturally have the same scaling.

When using likelihoods to compare between models it is necessary to leave the proportionality factors in their 'natural' or complete form, as @user75138 notes at the end of their answer.

Answer 5

For each factor in the likelihood (i.e. for each individual) "n" = $1$ and "x" = $0$ or $1$. In this case with ($n=1$) we always have $C_x = 1$. So $n C_x = 1$ for each of the factors making up the likelihood. So the normalization IS there, it is just $1$.

In general, a good check that one has written down the likelihood correctly and completely (i.e. including all of the factors, even if they do not affect an MLE calculation) is that if you sum the likelihood over all possible realizations of the data you get $1$. It is easy to see that the Miller and Freund's formula is normalized to $1$ this way (just sum over all $x_i = 0$ and $x_i =1$ for all of the i's, one gets $(1-p) + p = 1$ for each $i$ factor)