I've seen that the likelihood function of the parameter $\lambda$ of a Bernoulli variable given the training data $\boldsymbol X$ with labels $\boldsymbol y$ is modeled like this:

$$p(\boldsymbol y\mid\boldsymbol X;\lambda)=\prod\limits_{i=1}^I\lambda^{y_i}(1-\lambda)^{1-y_i}$$

I understand that if we consider the negative logarithm of this as a loss of an optimization problem, then the parameter is correctly adapted because if the ground truth label is $y_i=1$ then the exponent selects the first term and in order to maximize the likelihood it needs to be maximized (until it matches the ground truth label 1), and if $y_i=0$ the exponent selects $(1-\lambda)$, so in order to maximize the likelihood, $\lambda$ needs to be minimized (until it matches the ground truth label 0). Can this likelihood function be derived from first principles, e.g. using the fact that $\lambda$ is the mean of the distribution?


In essence you take the product because of the product rule for the probability of the intersection of independent events (i.e. $P(AB) = P(A) \, P(B)$, but here taken over more than two events).

See the corresponding rule for random variables; in the case of the Bernoulli we'd be using $f$ to represent the pmf rather than a density. That is just a consequence of the basic probability rule I mentioned in the first paragraph.

Then considering the definition of likelihood, the result follows immediately.

Suppressing the $X$ for clarity and using your $p$ rather than Wikipedia's $f$ to better emphasize the discreteness:

\begin{align} L(\lambda) & = p(\mathbf{y}\mid\lambda) & & _\text{(definition)} \\[10pt] & = p(y_1\mid\lambda)\times p(y_2\mid\lambda)\times \cdots \times p(y_n\mid\lambda) & & _\text{(product rule)} \end{align}

This is not just true for the Bernoulli of course; it's used for likelihood any time you have independence.

| cite | improve this answer | |
  • $\begingroup$ @LenarHoyt why what..? Why this is the formula for Bernoulli probability mass function? $\endgroup$ – Tim Feb 23 '17 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.