I am trying to learn a (neural) density estimator for a set of data p(x), however I know that the true distribution is a mixture of two other distributions, q(x) and z(x), with fixed mixture weight. The nature of the application requires me to learn p and not attempt to learn q independently.

Here's the twist: for z I already have a density model and I would like to make use of that fact to reduce the variance. Unfortunately, sampling from z is costly so I can't just use z to generate additional training samples.

It seems to me that if all my samples came from q, the best thing I could do was to perform maximum likelihood on the neural density model. And if all my samples came from z, the best thing to do would be to perform regression on the densities given by z. However, since I have both kinds of samples, it seems to me that I need a way to combine those two approaches.

What would be the best way to approach this problem?


Have one network output probabilities $\pi_q(x)$ and $\pi_z(x)$ (summing to 1) that sample $x$ "belongs" to component $q$ and $z$ of the mixture respectively.

Have another network to model $q(x)$. Then your final density is simply $\pi_q(x)q(x) + \pi_z(x)z(x)$, and you can train with max likelihood.

Train the assignment network and the $q$-density network simultaneously.

It wasn't clear to me whether you also wanted to replace your $z$-model with a neural component as well. If you do, you can replace the $z$-model with a neural model after the first part of the training has converged, and train everything jointly.

And of course if you want a single unified network for the mixture you can simply train another network to match whatever was learned by the $q$, $z$, and assignment networks.

| cite | improve this answer | |
  • $\begingroup$ I didn't want to go into any details but as I said, training q independently is impossible. (Basically, q and z are shifting and the objective is a moving target) $\endgroup$ – DaVinci Apr 29 '18 at 15:44
  • 1
    $\begingroup$ @DaVinci What do you mean by training independently? I am proposing a single model which will be trained on $x \sim p$, although within that model is a model for $q$ -- surely that should be ok? $\endgroup$ – shimao Apr 29 '18 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.