3
$\begingroup$

For reasons owing to mathematical convenience, when finding MLEs (maximum likelihood estimates), it is often the log-likelihood function---as opposed to the standard likelihood function---which is maximised.

From what I've gathered, this approach is deemed valid as a result of the monotonically increasing nature of the (natural) logarithm function.

My understanding of a monotonically increasing function is: for all $x$ and $y$ (defined on a subset of the reals), if $x \leq y$ then $f(x) \leq f(y)$.

This, however, does not appear to be the case for all log-likelihood functions; for example: for the log-likelihood $\text{Gamma}(3, 5)$ function, if $x = 0.15$ and $y = 0.46$, then $f(x) = -0.59$ and $f(y) = -1.19$.

Clearly, I've misunderstood this concept. Fundamentally, I guess I'm asking can somebody (preferably mathematically) demonstrate why:

$$ \hat{\theta} = \text{argmax} \text{ } L(\theta) = \text{argmax} \text{ } \text{log} \text{ } L(\theta) $$

where $\hat{\theta}$ is the MLE for a given likelihood function.

$\endgroup$
3
  • 5
    $\begingroup$ The problem is that the justification you use is not the correct one. It is not the likelihood function that must be monotonous but the transformation you apply to it. E.g. the result comes from the fact that for a concave likelihood function, the value of $\theta$ that maximizes $L(\theta)$ is the same as that which maximizes $\log L(\theta)$. This is a property of monotone transformations of concave functions $\endgroup$
    – user603
    Feb 25 '13 at 17:55
  • 6
    $\begingroup$ @user603 It's actually the most basic property of monotonic transformations, period: it follows directly from their definition and assumes nothing about $L$ itself. After all, if $L(\theta)\gt L(\theta')$ for all $\theta'$ (that is, $\theta$ maximizes $L$) and $x \gt y$ implies $f(x) \gt f(y)$ whenever $x$ and $y$ are both in the domain of $f$ (that is, $f$ is monotonic), then by letting $x=L(\theta)$ and $y=L(\theta')$ it is immediate that $f(L(\theta))\gt f(L(\theta')))$ for all $\theta'$ (that is, $\theta$ maximizes $f\circ L)$. $\endgroup$
    – whuber
    Feb 25 '13 at 18:32
  • 2
    $\begingroup$ Thanks for your assistance, everybody. As stated above, I had been under the impression that the likelihood function itself must be monotonic (not the transformation applied to it) which didn't seem to make sense. As @whuber states, the property follows directly from the definition of monotonic transformations. $\endgroup$
    – user9171
    Feb 26 '13 at 11:44
4
$\begingroup$

This is a direct consequence of properties of monotone (increasing) transformation, and the logarithm is monotone increasing. If there exist a value of $\theta$ that maximizes the likelihood function, that same value of $\theta$ will maximize the log likelihood function. The later is often preferred because it has better numerical properties, so is easier to maximize in practice. That is not the only reason, the log likelihood function arises also much in theory.

EDIT The answer by @David Grenier: shows this by using calculus. But calculus is not necessary! Let $f$ be a monotonous increasing function, like $\log$. This means that for all $x,y$, if $x\le y$ then $f(x) \le f(y)$. Let $\hat\theta$ be an MLE, so that for all $\theta$, $L(\theta)\le L(\hat\theta)$ where $L$ is the likelihood function.

Applying $f$ to both sides we can conclude $f(L(\theta))\le f(L(\hat\theta))$. Now, if $f$ (like $\log$) is strictly increasing, it has an inverse function, so the above argument can be reversed.

$\endgroup$
2
  • 1
    $\begingroup$ Pretty sure this is answered elsewhere on site. $\endgroup$
    – Glen_b
    Aug 24 '17 at 18:14
  • $\begingroup$ Probably true ... just search $\endgroup$ Aug 24 '17 at 18:15
1
$\begingroup$

Here's a simple proof that may help from a mathematical perspective. It's mostly just basic calculus.

Let $L(\theta)$ be a continuous function such that $L(\theta) > 0$ for some interval on the real number line (the support). Let $L(\theta)$ be maximized at $\hat{\theta}$.

If $L(\theta)$ is maximized at $\hat{\theta}$ then by the first and second derivative tests, $L^{\prime}(\hat{\theta}) = 0$ and $L^{\prime \prime}(\hat{\theta}) < 0$. Further as defined above we know $L(\hat{\theta}) > 0$. This gives us: $L(\hat{\theta}) > 0$; $L^{\prime}(\hat{\theta}) = 0$; $L^{\prime \prime}(\hat{\theta}) < 0$.

Let $\ell(\theta) = \ln[L(\theta)]$. Then $\ell(\theta) = (f \circ g)(\theta)$ where $f = \ln(x)$ and $g = L(x)$

If $\ell(\theta) = (f \circ g)(\theta)$ then by the chain rule $\ell^{\prime}(\theta) = (f' \circ g)g' = \frac{1}{L(\theta)} \cdot L^{\prime}(\theta) = \frac{L^{\prime}(\theta)}{L(\theta)}$

If $\ell^{\prime}(\theta) = \frac{L^{\prime}(\theta)}{L(\theta)}$ then by the quotient rule $\ell^{\prime \prime} = \frac{L^{''}L - L'L'}{L^2}$

If $\ell^{\prime}(\theta) = \frac{L^{\prime}(\theta)}{L(\theta)}$ and $L^{\prime}(\hat{\theta}) = 0$, then $\ell^{\prime}(\hat{\theta}) = \frac{0}{L(\hat{\theta})} = 0$ and $\ell(\hat{\theta})$ is a critical point.

If $\ell^{\prime \prime} = \frac{L^{''}L - L'L'}{L^2}$ and $L^{\prime}(\hat{\theta}) = 0$ then $\ell^{\prime \prime}(\hat{\theta}) = \frac{L^{''}(\hat{\theta})L(\hat{\theta})}{\left[L(\hat{\theta})\right]^2}$. Further, if $L(\hat{\theta}) > 0$ and $L^{\prime \prime}(\hat{\theta}) < 0$, then $\frac{L^{''}(\hat{\theta})L(\hat{\theta})}{\left[L(\hat{\theta})\right]^2} < 0$ and $\ell^{\prime \prime}(\hat{\theta}) < 0$

If $\ell^{\prime}(\hat{\theta}) = 0$ and $\ell^{\prime \prime}(\hat{\theta}) < 0$ then $\ell(\hat{\theta})$ is a minimum on the support of $L$.

Therefore if $L(\theta)$ is maximized at $\hat{\theta}$ then $\ln[L(\theta)]$ will also be maximized at $\hat{\theta}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.