Deep generative models learning a Bayesian-network distribution

Question

Say I have a generative model for some distribution $p$ over a small number of RVs which allows me to easily sample from said distribution. For example, let's say it's a parameterized Bayesian network $\mathcal{B}$ over $n$ random variables $X_1, \ldots, X_n$, which constrains the joint distribution $p_\mathcal{B}(X_1, \ldots, X_n)$ with the various conditional independencies which the BN structure implies. Given such a network I can easily generate as many samples $(x_1,\ldots,x_n)\sim p_\mathcal{B}$ as I want by forward sampling.

Question:

Is there any work on training a generative deep model to generate samples from such a distribution? (That is, to imitate the original generator), ideally with a meaningful latent space. Are there any generative models which are expected to do better/worse on such a task? I'm especially interested in the case where there are relatively few variables (no more than a few dozen) and the joint distribution is highly constrained (so probably can be mapped to from a simple latent space)

Answer 1

To answer your question first: No, I'm not aware of such work in the literature.

There are two classes of generative models which may be well-suited for this kind of modeling though.

1. VAE with an autoregressive (linear OR tree-structured decoder). Basically these types of models define $$p(x) = \int p(z)\prod_i p(x_i|x_{j < i},z) dz$$ And $z$ is the latent space. So each variable $x_i$ is generated conditioned on it's predecessors (in the linear generator case), or you can condition on it's parents or markov blanket depending on the exact structure of the bayesian network. The factors $p(x_i|x_{j < i},z)$ are modeled by an RNN, and just like in a typical VAE, we have a variational posterior $q(z|x)$ and the ELBO is optimized to train the model. The main problem with these types of models is that the decoder can become too powerful, and $q(z|x)$ just converges to $p(x)$ -- this is called "posterior collapse".

2. Deep EBMs -- these models use a network which directly outputs $E(x; \theta)$ and define $p(x) \propto \exp(-E(x; \theta))$. See here for an example. This energy function could even be parameterized with a graph neural network with the same graph as the bayesian network you're trying to copy. This type of model doesn't really have a latent space, but I suspect picking any pooling/bottleneck layer of the network would give good results.