17
$\begingroup$

Let's say we have $X_1,\ldots, X_n$ iid Bernoulli($p$), ask for MLE for $p$. I'm pretty struggled on the second derivative of log-likelihood function, why it is negative? My second question is what is MLE when the maximum is achieved on the boundary of the parameter space : ${\sum x_i}=0$ or $n$? Looking forward to any feedback and suggestions.

$\endgroup$
3
  • 1
    $\begingroup$ Well, I have another question which is how to prove $\frac{\sum x}{n}$ is global maximum rather an local maximum? $\endgroup$ Apr 23, 2017 at 18:30
  • $\begingroup$ Sorry for misleading statement,, my question was how to prove the second derivative is negative? $\endgroup$ Apr 23, 2017 at 20:29
  • 1
    $\begingroup$ Please edit the question to reflect your two comments so that people don't have to read the comments to understand what you want $\endgroup$
    – Glen_b
    Apr 24, 2017 at 10:01

2 Answers 2

25
$\begingroup$

Its often easier to work with the log-likelihood in these situations than the likelihood. Note that the minimum/maximum of the log-likelihood is exactly the same as the min/max of the likelihood. $$ \begin{align*} L(p) &= \prod_{i=1}^n p^{x_i}(1-p)^{(1-x_i)}\\ \ell(p) &= \log{p}\sum_{i=1}^n x_i + \log{(1-p)}\sum_{i=1}^n (1-x_i)\\ \dfrac{\partial\ell(p)}{\partial p} &= \dfrac{\sum_{i=1}^n x_i}{p} - \dfrac{\sum_{i=1}^n (1-x_i)}{1-p} \overset{\text{set}}{=}0\\ \sum_{i=1}^n x_i - p\sum_{i=1}^n x_i &= p\sum_{i=1}^n (1-x_i)\\ p& = \dfrac{1}{n}\sum_{i=1}^n x_i\\ \dfrac{\partial^2 \ell(p)}{\partial p^2} &= \dfrac{-\sum_{i=1}^n x_i}{p^2} - \dfrac{\sum_{i=1}^n (1-x_i)}{(1-p)^2} \end{align*} $$

The penultimate line gives us the MLE (the $p$ that satisfies the first derivative of the log-likelihood (also called the score function) equal to zero).

The last equation gives us the second derivative of the log-likelihood. Since $p\in [0,1]$ and $x_i \in \left\{0,1\right\}$, the second derivative is negative.

$\endgroup$
6
$\begingroup$

The negative sign of the second derivative shows that the stationary point is a maximum. A positive would indicate a minimum.

The second derivative tells you how the first derivative (gradient) is changing. A negative value tells you the curve is bending downwards. This occurs at a maximum.

Assuming from your post you already have the first derivative of the log-likelihood function \begin{equation} \frac{d\ \ln f}{dp}=\frac{\sum_i x_i}{p}-\frac{n-\sum_i x_i}{1-p} \end{equation}

giving \begin{equation} \hat{p}=\frac{\sum_i x_i}{n} \end{equation}

Second deriative \begin{equation} \frac{d^2(\ln f)}{dp^2}=-\frac{\sum_i x_i}{p^2}-\frac{n-\sum_i x_i}{(1-p)^2} \end{equation}

This will be negative for all real values of p, as inspection confirms that for the two fractions in the expression, both numerators and both denominators are positive.

We can demonstrate this for the specific value of $\hat{p}$ found.

Substituting your value of $\hat{p}$

\begin{equation} \frac{d^2(\ln f)}{d p^2}=-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(1-\frac{\sum_i x_i}{n})^2} \end{equation}

\begin{equation} =-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(\frac{n-\sum_i x_i}{n})^2} \end{equation} \begin{equation} =-\frac{n^2}{\sum_i x_i}-\frac{n^2}{n-\sum_i x_i} \end{equation}

Which is clearly negative.

You know this is a global maximum, as it is the only maximum! Minimums​ occur at the boundaries.

You could prove $p = 0$ was the maximum on the boundary by showing the gradient was always negative.

Likewise if gradient is always positive, this would prove $p = 1$ is the maximum.

$\endgroup$
6
  • $\begingroup$ Sorry for misleading statement,, my question was how to prove the second derivative is negative? $\endgroup$ Apr 23, 2017 at 20:30
  • 1
    $\begingroup$ That makes more sense. I'm​ typing on my phone, so i won't attempt to write equations. The steps will be to find second derivative, then substitute value of p that gives stationary point. $\endgroup$ Apr 23, 2017 at 20:45
  • $\begingroup$ Back to my computer! I've added second derivative. $\endgroup$ Apr 24, 2017 at 2:46
  • $\begingroup$ Your argument is incorrect: even when only one maximum can be found among the critical points of a function, it is not necessarily a global maximum. Counterexample: $y=x^4-x^2$ has a single maximum at $x=0$ among its critical points, but its value ($0$) clearly is not a global maximum. $\endgroup$
    – whuber
    Jan 18, 2019 at 20:33
  • $\begingroup$ @whuber I think the argument is valid because in this case the optimization is bounded on both sides. In bounded optimization, you are guaranteed to find global optima by checking all critical points plus the boundary. $\endgroup$ Nov 15, 2019 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.