5
$\begingroup$

Let's say we have $X_1,\ldots, X_n$ iid Bernoulli($p$), ask for MLE for $p$. I'm pretty struggled on the second derivative of log-likelihood function, why it is negative? My second question is what is MLE when the maximum is achieved on the boundary of the parameter space : ${\sum x_i}=0$ or $n$? Looking forward to any feedback and suggestions.

$\endgroup$
  • $\begingroup$ Well, I have another question which is how to prove $\frac{\sum x}{n}$ is global maximum rather an local maximum? $\endgroup$ – keqiao li Apr 23 '17 at 18:30
  • $\begingroup$ Sorry for misleading statement,, my question was how to prove the second derivative is negative? $\endgroup$ – keqiao li Apr 23 '17 at 20:29
  • 1
    $\begingroup$ Please edit the question to reflect your two comments so that people don't have to read the comments to understand what you want $\endgroup$ – Glen_b Apr 24 '17 at 10:01
12
$\begingroup$

Its often easier to work with the log-likelihood in these situations than the likelihood. Note that the minimum/maximum of the log-likelihood is exactly the same as the min/max of the likelihood. $$ \begin{align*} L(p) &= \prod_{i=1}^n p^{x_i}(1-p)^{(1-x_i)}\\ \ell(p) &= \log{p}\sum_{i=1}^n x_i + \log{(1-p)}\sum_{i=1}^n (1-x_i)\\ \dfrac{\partial\ell(p)}{\partial p} &= \dfrac{\sum_{i=1}^n x_i}{p} - \dfrac{\sum_{i=1}^n (1-x_i)}{1-p} \overset{\text{set}}{=}0\\ \sum_{i=1}^n x_i - p\sum_{i=1}^n x_i &= p\sum_{i=1}^n (1-x_i)\\ p& = \dfrac{1}{n}\sum_{i=1}^n x_i\\ \dfrac{\partial^2 \ell(p)}{\partial p^2} &= \dfrac{-\sum_{i=1}^n x_i}{p^2} - \dfrac{\sum_{i=1}^n (1-x_i)}{(1-p)^2} \end{align*} $$

The penultimate line gives us the MLE (the $p$ that satisfies the first derivative of the log-likelihood (also called the score function) equal to zero).

The last equation gives us the second derivative of the log-likelihood. Since $p\in [0,1]$ and $x_i \in \left\{0,1\right\}$, the second derivative is negative.

$\endgroup$
4
$\begingroup$

The negative sign of the second derivative shows that the stationary point is a maximum. A positive would indicate a minimum.

The second derivative tells you how the first derivative (gradient) is changing. A negative value tells you the curve is bending downwards. This occurs at a maximum.

Assuming from your post you already have the first derivative of the log-likelihood function \begin{equation} \frac{d\ \ln f}{dp}=\frac{\sum_i x_i}{p}-\frac{n-\sum_i x_i}{1-p} \end{equation}

giving \begin{equation} \hat{p}=\frac{\sum_i x_i}{n} \end{equation}

Second deriative \begin{equation} \frac{d^2(\ln f)}{dp^2}=-\frac{\sum_i x_i}{p^2}-\frac{n-\sum_i x_i}{(1-p)^2} \end{equation}

This will be negative for all real values of p, as inspection confirms that for the two fractions in the expression, both numerators and both denominators are positive.

We can demonstrate this for the specific value of $\hat{p}$ found.

Substituting your value of $\hat{p}$

\begin{equation} \frac{d^2(\ln f)}{d p^2}=-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(1-\frac{\sum_i x_i}{n})^2} \end{equation}

\begin{equation} =-\frac{\sum_i x_i}{(\frac{\sum_i x_i}{n})^2}-\frac{n-\sum_i x_i}{(\frac{n-\sum_i x_i}{n})^2} \end{equation} \begin{equation} =-\frac{n^2}{\sum_i x_i}-\frac{n^2}{n-\sum_i x_i} \end{equation}

Which is clearly negative.

You know this is a global maximum, as it is the only maximum! Minimums​ occur at the boundaries.

You could prove $p = 0$ was the maximum on the boundary by showing the gradient was always negative.

Likewise if gradient is always positive, this would prove $p = 1$ is the maximum.

$\endgroup$
  • $\begingroup$ Sorry for misleading statement,, my question was how to prove the second derivative is negative? $\endgroup$ – keqiao li Apr 23 '17 at 20:30
  • 1
    $\begingroup$ That makes more sense. I'm​ typing on my phone, so i won't attempt to write equations. The steps will be to find second derivative, then substitute value of p that gives stationary point. $\endgroup$ – Jeremy Voisey Apr 23 '17 at 20:45
  • $\begingroup$ Back to my computer! I've added second derivative. $\endgroup$ – Jeremy Voisey Apr 24 '17 at 2:46
  • $\begingroup$ Your argument is incorrect: even when only one maximum can be found among the critical points of a function, it is not necessarily a global maximum. Counterexample: $y=x^4-x^2$ has a single maximum at $x=0$ among its critical points, but its value ($0$) clearly is not a global maximum. $\endgroup$ – whuber Jan 18 at 20:33
  • $\begingroup$ @whuber I think the argument is valid because in this case the optimization is bounded on both sides. In bounded optimization, you are guaranteed to find global optima by checking all critical points plus the boundary. $\endgroup$ – Radon Rosborough 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.