How to choose a way to obtain maximum likelihood estimation

Question

Given an example random variables $ X_1, X_2,\ldots, X_n $, and assume that we have a distribution function to describe the data (Bernoulli):

$f(x_i;p) = p^{x_i}(1-p)^{(1-x_{I})} $

Here $p$ is the unknown parameter, with which we want to find maximum likelihood estimator (MLE).

In my understanding there are two ways to obtain that MLE:

1. Is by using calculus, do it on paper like this way. This lead us to obtain $\hat{p} = \frac{\sum_{i=1}^n X_i}{n}$. With which then we can just plug-in the $ X_1, X_2,\ldots, X_n $ value and get the actual value for $\hat{p}$.

2. Numerical (iterative) search procedure like using bbmle::mle2 package in R.

My question, given a problem how can we decide which method to use to find MLE. With calculus or numerically search procedure?

Answer 1

There is nothing special about MLEs here, since what you're asking really applies to any optimisation problem. In this broader sense, you want to know what methods are desirable to maximise/minimise a function.

Applying standard calculus methods for optimisation, you should be able to get an equation for the critical points for the function, and the second-order-conditions for optimisation, and this will usually allow you to find the maximising point. Often the maximising point is a critical-point, and in this case, optimisation involves finding the solution(s) to the critical point equation. In some problems the critical point has an explicit closed-form solution, as in the example in your question. In other problems there is no closed-form solution to the critical point equation, and in this case it is usual to solve this equation via iterative numerical methods (i.e., start at some arbitrary point, and then move closer and closer to the critical point via some iterative algorithm).

The best way to distinguish these cases is simply to try to optimise algebraically, and see if this leads you to an explicit solution, or a case that requires numerical methods. If it is possible to get an explicit closed-form solution for the maximising value then you are finished, and no numerical methods are required. If you require numerical methods to find the solution to the critical point equation, then you can either form this iterative process algebraically, or rely on a package such as bbmle in R. At the end of the day, optimisation requires familiarity with calculus; with experience you will be able to anticipate which problems give closed-form solutions and which will require iterative methods.