I am trying to learn a (neural) density estimator for a set of data p(x), however I know that the true distribution is a mixture of two other distributions, q(x) and z(x), with fixed mixture weight. The nature of the application requires me to learn p and not attempt to learn q independently.
Here's the twist: for z I already have a density model and I would like to make use of that fact to reduce the variance. Unfortunately, sampling from z is costly so I can't just use z to generate additional training samples.
It seems to me that if all my samples came from q, the best thing I could do was to perform maximum likelihood on the neural density model. And if all my samples came from z, the best thing to do would be to perform regression on the densities given by z. However, since I have both kinds of samples, it seems to me that I need a way to combine those two approaches.
What would be the best way to approach this problem?