MLE over a large number of parameters

Question

I am working on a model that I am fitting to a medium-sized dataset ($\approx 300,000$ observations). The likelihood I have derived is a mixture model of some Beta distributions which also has the convolution of two Betas. This makes the evaluation of the log-likelihood function slow. The likelihood function has nine variables I am optimizing over. This makes even a coarse grid search very computationally slow, even if I parallelize it. A grid search of just four values per parameter requires $4^9=262,144$ evaluations of the log-likelihood, and still does not give particularly great starting parameters for the optimization. I then use the best set of parameters from the grid search as my starting guess in the actual optimization (optim in R using the L-BFGS-B method as many parameters need to be non-negative or bounded between 0 and 1). Does anyone know of a better and/or faster way to do an MLE over a large number of parameters?

Answer 1

Consider subsampling your observations, you dont need to use all 300,000 observations to estimate 9 parameters. That is an incredibly large amount of data for a fairly low dimensional parameter vector. If you picked a sub-sample of (say) 10,000 observations then you could minimise this partial likelihood instead which will be a lot faster, and the minimum is probably going to be close to the true minimum using all the data (so you can then use the minimum you find as an initial value for the full minimisation using all the data). If you do this using multiple sub-samples then it might even help you find different local minimas (but are you sure that your likelihood is really highly multimodal? Because with 300,000 observations and only 9 parameters I would expect it to be very sharply peaked).

Better yet, just use stochastic gradient descent instead, where you process the observations one-at-a-time rather than in batch: https://en.wikipedia.org/wiki/Stochastic_gradient_descent

Answer 2

Unless for trivial problems, of simple hyperparameter tuning, nobody uses grid search as an optimization algorithm. Mixture of beta distributions is parametrized by continuous parameters, so there is infinitely many values of the parameters, and their combinations, that you would need to check. This would not end in finite time if using grid search.

There is a whole branch of mathematics devoted to optimization and a number of algorithms for optimizing different functions. In R there are build-in optim and optimize functions, while in python there is scipy.optimize.minimize that provide the most popular optimization algorithms. Those algorithms take the function that you want to minimize and find the values for you. However, please take time to read the documentation first and read about the algorithms, since those functions implement many different algorithms, where some are known to work better for some classes of problems, then others.

As for mixture, the most popular algorithm for fitting mixtures is E-M algorithm, in fact, the linked Wikipedia page uses Gaussian mixture as an example illustrating the algorithm.

Since you do not seem to be familiar with optimization, I'd recommend to start with learning more about it. There's a handbook Algorithms for Optimization by Kochenderfer and Wheeler and recorded Stanford lectures by Stephen Boyd, that could serve as a nice introduction.