Timeline for MLE over a large number of parameters
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 24, 2020 at 19:05 | comment | added | James | To turn the question around, if the OP had 300 billion observations rather than 300,000, and the (complete data) likelihood took over a day to evaluate would you accept that batch methods like EM are completely infeasible and that stochastic techniques are the more sensible approach? The only question is the cross-over point at which stochastic algorithms are likely to be better and I would cautiously suggest that 300,000 observations is likely to be above that point if the likelihood is complicated. But really the OP needs to just try it and see how it works, there are R packages. | |
| Jun 24, 2020 at 18:55 | comment | added | James | papers.nips.cc/paper/… arxiv.org/pdf/0712.4273.pdf (note that these are both discussing stochastic vs batch EM. If I was the OP personally then I'd first try vanilla SGD to see how well it performed since its trivial to implement, and only think about implementing stochastic EM if it didnt work well. But with 300000 observations I'd definitely be using some kind of stochastic/minibatch algorithm, if the likelihood is taking too long to evaluate) | |
| Jun 24, 2020 at 18:54 | comment | added | James | "Standard" applications of mixture models dont have 300,000 observations. Stochastic gradient descent is a standard technique for optimising parameters over large data sets since it uses the data more efficiently than batch gradient descent algorithms like EM (since it is does not 'waste time' computing quantities to unneeded levels of precision). Batch algorithms typically dont scale well to large data sets for this reason. Here are some specific discussions of SGD in a mixture model framework, but really this is part of a much larger discussion on batch vs stochastic algorithms | |
| Jun 24, 2020 at 17:51 | comment | added | Tim | EM is a standard approach to mixture models, SDG not (see e.g. stats.stackexchange.com/questions/64193/…), so it’d help if you could give some references showing the superiority you mention. | |
| Jun 24, 2020 at 17:06 | comment | added | James | Finally the inherent multimodality of mixture models typically isnt a problem in this context since these multiple modes are just relabellings of the mixture components and hence give equivalent likelihoods. Finding any of these modes is as good as finding any of the others (there may be other modes of course, but with 300,000 observations I would expect these to be largely smoothed out so the main multimodality would just be due to the above label/component index switching) | |
| Jun 24, 2020 at 17:06 | comment | added | James | There is no reason why EM would be superior to stochastic gradient descent in a mixture model context when the number of observations is large, since so much computational effort in EM is being 'wasted' by using the full set of observations to produce overly accurate estimates of quantities which will already be precisely estimated using only a small subset of the data. As such, SGD will be able to run hundreds of thousands of iterations in the time it takes EM to run a single iteration. I would expect SGD to perform better for this reason, but its data-set dependent. | |
| Jun 24, 2020 at 16:57 | comment | added | James | Hi Tim, I think you misunderstood my suggestion. The point of using subsampling is to get a good initial value for the full minimisation problem. I.e. to find an estimate of theta which is likely to be close to the true minimum, and then use this as the initial condition for optimisation when using the full 300,000 observations. | |
| Jun 24, 2020 at 6:25 | comment | added | Tim | Moreover, yes, mixture models are multimodal by design, there is no single true minimum. Also SGD is not the best choice of algorithm. First of all, it is not the most efficient optimization algorithm in general. Second, for mixtures there are specialised algorithms (like E-M) that are known to work better. | |
| Jun 24, 2020 at 6:23 | comment | added | Tim | I don't think this is good set of advice for this problem. First of all, no need to subsampling. With subsampling you loose information, sure that you can estimate 9 parameters with far less samples (you could go all the way down to 9+1), but the more samples, the more precise the result will be. 300k parameters should with no problem fit RAM of desktop PC & the computation time should take not more then minutes, so no reason why the size should be problem. | |
| Jun 24, 2020 at 2:21 | history | edited | James | CC BY-SA 4.0 |
deleted 22 characters in body
|
| Jun 24, 2020 at 2:16 | history | answered | James | CC BY-SA 4.0 |