I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are assumed to come from a mixture distribution
$$ p(x_i) = \sum_{k=1}^K \pi_k \cdot \mathcal N(x_i | \mu_k, \sigma_k^2). $$
I wish to estimate both the number of components $K$ and the parameters $\pi_k, \mu_k, \sigma_k^2$ in an online (or pseudo-online) fashion. In actuality I also wish to allow for the possibility that $K$ and the parameters may change on long time scales.
My current, somewhat ad hoc, solution is to maintain a window of the $N = 1000$ most recent observations, and reestimate the model every $n = 100$ observations (using the windowed data). For each $K = 1, 2, \dotsc, 10$ I run expectation maximization (EM) with a prior to (hopefully) find the MAP estimate. (The prior helps to avoid the singularity issues of ML for Gaussian mixtures.) Then I use the Bayesian information criterion (BIC) to pick the "best" model of those 10.
This solution works well enough in practice, but the training is a bit slow and it has a somewhat large memory footprint. We run many instances of this algorithm, so these things add up. I would much prefer to have a fully online algorithm.
I briefly considered using variational inference for the mixture. This would have the advantage that I could simply set $K$ to the maximum value of 10, since a variational mixture will tend to supress superfluous components automatically. But it would still not be online.
I searched a little bit for online versions of expectation maximization, but everything I found seemed pretty complex. Is there a good alternative that I am overlooking?
