1
$\begingroup$

I always found that in mathematical statistics books etc, they would state that to find the maximum we take the logarithm of the function and then differentiate, set to zero and get the Maximum value? I have not seen a proof as to why it would give a Maximum, why not Minimum?

Here is from a tutorial on mathematical stats online : "maximum value of the log of the probability occurs at the same point as the original probability function"

People state this without proof. Hope some one can show me proof of this assertion.

Note, my question is not a duplicate, because in the other link, it still does not give proof. And as Dave below says " We assume maximum without proof via second derivative". And that is my question, that is not answered in any of the other links, There is still NO proof, just assertions that it is true WITHOUT PROOF!

$\endgroup$
11
  • 1
    $\begingroup$ Are you confused about why taking the logarithm allows us to calculate the maximum of the original likelihood function, why we assume a maximum without taking a second derivative, or both? $\endgroup$
    – Dave
    Jun 28, 2019 at 2:42
  • $\begingroup$ I just looked at this, link you provided Sycorax, and i don't see a Proof happening in this link. It looks like people are just talking in a kind of speculation way. $\endgroup$
    – Palu
    Jun 28, 2019 at 2:48
  • $\begingroup$ Hi Dave, I am especially interested in this what you said: "why we assume a maximum without taking a second derivative" $\endgroup$
    – Palu
    Jun 28, 2019 at 2:49
  • 1
    $\begingroup$ I urge you to re-read the duplicate thread, particularly whuber's comment about monotonic transformations. I think you'll find your answer immediately. $\endgroup$
    – Sycorax
    Jun 28, 2019 at 3:03
  • $\begingroup$ Hi Sycorax, if you mean this "Thanks for your assistance, everybody. As stated above, I had been under the impression that the likelihood function itself must be monotonic (not the transformation applied to it) which didn't seem to make sense. As @whuber states, the property follows directly from the definition of monotonic transformations"--------------------------------------------------------------------- I still don't get it. It is too cryptic if this is supposed to be a proof. The proof must be in such a way that a student that has highschool Calculus can understand it. $\endgroup$
    – Palu
    Jun 28, 2019 at 3:05

2 Answers 2

3
$\begingroup$

Let's say that you find a unique critical point of the log-likelihood function. This need not be the global maximum of the function. It need not even be a maximum! The maximum could occur at the boundaries, perhaps $\pm \infty$ on $\mathbb{R}$ or $0$ and $1$ on $[0,1]$ for a Bernoulli distribution. So it is necessary to check the boundaries. If the boundaries are lower than the critical point, you can play some games with the intermediate value theorem to deduce that your critical point is the maximum. Now you have your MLE!

The likelihood function need not be so nice that you can get away with tricks like these, however, and you may find yourself in a position where it's necessary to take second derivatives.

Concerning why we take the logarithm, the likelihood function ends up being a big product. For $X_1,\dots,X_n$, we multiple $n$ functions. That's a very nasty product rule derivative. Taking the logarithm turns the multiplication into addition: $log(xy) = log(x) + log(x)$. The differentiation is much easier. The maximum of $f(x)$ is the same as the maximum of $log(f(x))$, so we prefer the function with the easier derivative. We don't care what the maximum of the likelihood function is. We care about where it occurs, as that is the MLE.

$\endgroup$
6
  • $\begingroup$ Yes Dave, thanks. That is a fairly good explanation. BUT i am trying to see if this generalizes beyond this "Maximum Likelihood Estimation stuff", the question is, is this a general method to find the x-value of the maximum for any continuous single variable real-valued function. $\endgroup$
    – Palu
    Jun 28, 2019 at 3:11
  • $\begingroup$ The thing that bothers me is that all the books just state this without proof and also give the implied feeling that this is a universal method for finding maximums. If it is the case then they should have taught this in the first course in Calculus. Seems to me to be some kind of trick that statisticians don't want to fully explain. $\endgroup$
    – Palu
    Jun 28, 2019 at 3:13
  • $\begingroup$ Ok, so you are saying that with intermediate value theorem to deduce the critical point is a maximum. Many professors out there when they go into this topic just tell the students to take the logarithm, differentiate, set to zero and the critical number is the maximum. I have recently helped students out with this, but it always bothered me that no Prof has done a decent job to explain it fully, properly. And the books are even worse in general on this. SO this topic really gets to me! $\endgroup$
    – Palu
    Jun 28, 2019 at 3:42
  • $\begingroup$ So, thanks Dave i will mark your answer as the official answer to this question, and i did not know about checking endpoints on this kind of problem. $\endgroup$
    – Palu
    Jun 28, 2019 at 3:43
  • $\begingroup$ That's usually enough to get the MLE. Someone might know a proof about concavity for certain distributions (exponential family, perhaps) where this is certain to be acceptable, which it could be argued is acceptable to gloss over in a mild introduction to MLEs. PS: Thanks for the check mark! $\endgroup$
    – Dave
    Jun 28, 2019 at 3:46
0
$\begingroup$

Paninski, 2004 : https://www.cns.nyu.edu/pub/eero/paninski04b.pdf

Basically we take the logarithm for easier calculations but in most of cases, since the function is increasing and the second derivative is negative, you have a global maximum.

$\endgroup$
4
  • $\begingroup$ Hi, thanks for both your responses. To Sycorax, i don't think its exactly a duplicate. To josef_joestarr, we don't know if a function is increasing when given a probability function in general, and if it was increasing how does that illustrate that it would have a maximum. BUT also if we take the simplest quadratic that has a maximum like y = -x^2, taking the log-based method would fail in my opinion, so i want to say that i don't think this is a general method that handles all real valued functions.(i am avoiding complex numbers) $\endgroup$
    – Palu
    Jun 28, 2019 at 2:46
  • $\begingroup$ @ Palu $-x^2$ is maximized at $x=0$. Take the logarithm to get $log(-x^2) = log(1) - log(x^2) = -2log(x)$, which is maximized as $x$ approaches zero. $\endgroup$
    – Dave
    Jun 28, 2019 at 3:19
  • $\begingroup$ Hi Dave, I don't think your logarithm identity is correct. Recall Log(A/B) = Log(A) - Log(B). Also Log(AB) = Log(A) + Log(B). your log(1) - log(x^2) = log(1/(x^2)). So your use of this its not correct $\endgroup$
    – Palu
    Jun 29, 2019 at 18:15
  • $\begingroup$ I have no idea what I was thinking when I posted that. Anyway, I think the example of -x^2 isn’t a good example for a log transformation, since we take logs of positive numbers, but take the log of the function for perhaps the normal distribution to get a sense of how the maximizing input is the same with and without the log. $\endgroup$
    – Dave
    Jul 1, 2019 at 2:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.