I have a 3D dataset with at least millions of data points (scatter events from atoms, approximately Gaussian). I am modeling this data with a Gaussian Mixture Model. The usual approach would be to optimize the GMM with the EM algorithm. However, I have substantial priors (e.g., the data is produced by a polymer made of light atoms) so the only way to find the correct GMM is via sampling.
Here's the problem: it's very expensive to compare a GMM to the data $D$, and I have to do that evaluation frequently. I would like to introduce an intermediate step: first model $G_d$ without priors, and then introduce priors and re-model $G_m$ where both are GMMs. Then while sampling $G_m$ I could just compare it to $G_d$ instead of $D$, which is a lot cheaper. So $G_d$ is kind of a data reduction. Intuitively the likelihood function for $G_m$ should be:
$$ p(D|G_m)\approx p(D|G_d)S(G_d|G_m) $$
where $S(G_d|G_m)$ is some kind of scoring function (maybe K-L divergence? correlation?). My questions are:
- Can you re-model in this way and still get a good approximation of the likelihood?
- What types of functions can be used for $S(G_d|G_m)$ and still get a proper PDF?
- Does the choice of number of components in $G_d$ affect the final likelihood?
Below I attempt to illustrate this problem. The magenta $G_d$ is computed at the beginning. Then the red $G_m$ with its additional priors (including size and linearity) are fit to the magenta one, much faster than comparing to the data itself. Quick clarification: the number of Gaussians in $G_m$ is much higher than $G_d$. That's because the only prior on $G_d$ is the Dirichlet one (so it's just regular Bayesian GMM fitting) whereas $G_m$ has much more detailed priors which enables us to make a more detailed model than the data itself would indicate.