Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a huge dataset of features on which I want to fit a Gaussian Mixture Model using standard expectation maximization, as it is implemented by sklearn. Since not all features fit into the memory at once, I need to train my GMM iteratively, i.e. one batch after the other. What is the best approach to do this? Just go for GMM.fit(batch) for each batch? Maybe use a smaller number of iterations for the EM algorithm? Anymore thoughts, ideas, or even better references therefore?
Thanks!
This question has been asked on Stackoverflow a while back .
Since scikit-learn did not extend their implementation of Gaussian mixtures with a partial_fit method (as was the case with other models in the past) I would suggest applying the solution in the accepted answer - sample a subset of the data that fits into the available memory and learn the model for that.
Given the sample isn't too small, it should represent the original distribution well enough.
