Maximum Likelihood Estimation for Bernoulli distribution

Question

Let's say we have $X_1,\ldots, X_n$ iid Bernoulli($p$), ask for MLE for $p$. I'm pretty struggled on the second derivative of log-likelihood function, why it is negative? My second question is what is MLE when the maximum is achieved on the boundary of the parameter space : ${\sum x_i}=0$ or $n$? Looking forward to any feedback and suggestions.

Answer 1

Its often easier to work with the log-likelihood in these situations than the likelihood. Note that the minimum/maximum of the log-likelihood is exactly the same as the min/max of the likelihood. $$ \begin{align*} L(p) &= \prod_{i=1}^n p^{x_i}(1-p)^{(1-x_i)}\\ \ell(p) &= \log{p}\sum_{i=1}^n x_i + \log{(1-p)}\sum_{i=1}^n (1-x_i)\\ \dfrac{\partial\ell(p)}{\partial p} &= \dfrac{\sum_{i=1}^n x_i}{p} - \dfrac{\sum_{i=1}^n (1-x_i)}{1-p} \overset{\text{set}}{=}0\\ \sum_{i=1}^n x_i - p\sum_{i=1}^n x_i &= p\sum_{i=1}^n (1-x_i)\\ p& = \dfrac{1}{n}\sum_{i=1}^n x_i\\ \dfrac{\partial^2 \ell(p)}{\partial p^2} &= \dfrac{-\sum_{i=1}^n x_i}{p^2} - \dfrac{\sum_{i=1}^n (1-x_i)}{(1-p)^2} \end{align*} $$

The penultimate line gives us the MLE (the $p$ that satisfies the first derivative of the log-likelihood (also called the score function) equal to zero).

The last equation gives us the second derivative of the log-likelihood. Since $p\in [0,1]$ and $x_i \in \left\{0,1\right\}$, the second derivative is negative.

Answer 2

The negative sign of the second derivative shows that the stationary point is a maximum. A positive would indicate a minimum.

The second derivative tells you how the first derivative (gradient) is changing. A negative value tells you the curve is bending downwards. This occurs at a maximum.

Assuming from your post you already have the first derivative of the log-likelihood function \begin{equation} \frac{d\ \ln f}{dp}=\frac{\sum_i x_i}{p}-\frac{n-\sum_i x_i}{1-p} \end{equation}

giving \begin{equation} \hat{p}=\frac{\sum_i x_i}{n} \end{equation}

Second deriative \begin{equation} \frac{d^2(\ln f)}{dp^2}=-\frac{\sum_i x_i}{p^2}-\frac{n-\sum_i x_i}{(1-p)^2} \end{equation}

This will be negative for all real values of p, as inspection confirms that for the two fractions in the expression, both numerators and both denominators are positive.

We can demonstrate this for the specific value of $\hat{p}$ found.

Substituting your value of $\hat{p}$

\begin{equation} \frac{d^2(\ln f)}{d p^2}=-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(1-\frac{\sum_i x_i}{n})^2} \end{equation}

\begin{equation} =-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(\frac{n-\sum_i x_i}{n})^2} \end{equation} \begin{equation} =-\frac{n^2}{\sum_i x_i}-\frac{n^2}{n-\sum_i x_i} \end{equation}

Which is clearly negative.

You know this is a global maximum, as it is the only maximum! Minimums​ occur at the boundaries.

You could prove $p = 0$ was the maximum on the boundary by showing the gradient was always negative.

Likewise if gradient is always positive, this would prove $p = 1$ is the maximum.