1
$\begingroup$

In proofs of maximising log likelihood functions, the partial derivative of the log likelihood is taken with respect to the value we want to maximise the likelihood of estimating, and then this partial derivative result is set equal to 0 and solved for the value of interest.

At this point, why is the partial derivative set equal to $0$? Coming from a mathematics background, this is not how you find the maximum -- it is how you find the critical values.

I would greatly appreciate it if people could please clarify this.

Example given below for the maximum likelihood estimator of $\sigma^2$ for a simple linear regression:

enter image description here

$\endgroup$
4
  • 1
    $\begingroup$ The maximum must occur at a critical value, and the log-likelihood for linear and logistic regression have only one critical value. $\endgroup$ Mar 15, 2018 at 13:28
  • $\begingroup$ @MatthewDrury Thanks for the response. Can you please elaborate on why it only has one critical value? And it's not just the logistic regression -- I've seen the same thing with estimating parameters for normal and binomial random variables. $\endgroup$ Mar 15, 2018 at 13:30
  • 2
    $\begingroup$ @ThePointer alas not all likelihoods have one critical value. There are very complicated or very time-intensive techniques for those cases, but that's besides the point. Regular exponential families (like Normal, Poisson, Binomial, etc.) typically enjoy this property of existence/uniqueness of a maximum--and non-existence of a minimum. For these families, the "parameter" matches our intuition that as possible-parameters are evaluated farther from the MLE they should as likely or less likely. This "concavity" property makes estimation a breeze. $\endgroup$
    – AdamO
    Mar 15, 2018 at 13:52
  • $\begingroup$ @AdamO I understand now. Thank you for the clarification. $\endgroup$ Mar 15, 2018 at 13:54

1 Answer 1

1
$\begingroup$

To put @MatthewDrury's answer less correct but perhaps simpler;

As in high-school math, you find the maximum (or the minimum) of a function by setting it's derivative equal to zero. Here, it's multivariate, there are more parameters involved, but the idea is exactly the same. Find the combination of parameters for which the function has derivative zero, that's where the maximum or minimum might be found.

You can look at the examples at https://www.mathsisfun.com/calculus/maxima-minima.html.

It's not always that easy, this is only part of the work if the function has multiple maxima or minima for example. But in this case, there is exactly one combination of parameters that maximimizes the likelihood. In linear regression, that is the same parameter combination as the least squares estimate.

$\endgroup$
3
  • $\begingroup$ I see what you mean: blogs.sas.com/content/iml/files/2017/06/loglikcreate1.png $\endgroup$ Mar 15, 2018 at 13:42
  • $\begingroup$ Exactly, good picture as well. $\endgroup$
    – Gijs
    Mar 15, 2018 at 13:43
  • 3
    $\begingroup$ Setting derivatives to zero is only a part of the process of finding a maximum--and this distinction goes to the heart of the question, IMHO. The critical values also include the boundary of the parameter space as well as any points where the objective function is not differentiable. $\endgroup$
    – whuber
    Mar 15, 2018 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.