Are Deep Neural Nets Graphical Models?

In the talk, here at NIPS, they say that:

GANs and VAEs are Graphical Models, just with a particular CPD and cost function. They are bipartite complete graphs.

How can that be explained? I thought that you need probabilities enmeshed in the models, with variables having dependence relationships. In neural nets, they have all sorts of other things like ReLu nodes etc. i.e there are no probability relationships just a series of non linearities alongwith with priors such as regularization or convnet structure.

This seems to be very different view than that is explained at What's the relation between hierarchical models, neural networks, graphical models, bayesian networks? .


If you focus on the generative part, GANs and VAEs are actually mathematically the same object (1), i.e. Gaussian latent variable models, where $z$ is a latent Gaussian random variable pointing to an observed $x$:

enter image description here The difference is that VAEs are prescribed models that output a random variable $x$ with a probability density, while GANs are likelihood-free implicit models (2) that directly specifies a (deterministic) procedure with which to generate data.

graphical models Concretely, the VAE's graphical model is implemented as the decoder/inference network, while the GAN's graphical model is implemented as the generator network; the GAN's discriminator network does not appear in the graphical model (similarly to how the VAE's encoder/recognition network doesn't show up) because it is merely an auxiliary object created to approximate the Jensen-Shannon divergence or other f-divergences (1):

GANs vs VAEs (Image from Lilian Weng)


(1): I've linked the relevant timestamp of the video recording of a tutorial (slides here) by Shakir Mohamed and Danilo Rezende from DeepMind at UAI 2017; Ferenc Huszar also explains the equivalence on Reddit. The VAE's graphical model is also explained in Stanford's "CS236 Deep Generative Models" notes.

(2): The distinction between prescribed and implicit models is described in greater detail in "Learning in Implicit Generative Models" by Shakir et al. (2016).

Extra: Another perspective with augmented graphical models

On Unifying Deep Generative Models (Hu et al., 2017) illustrates augmented graphical models that encompass both the GAN generator and discriminator, and an analogous model for the VAE where we assume a perfect discriminator.

GAN and VAE graphical model

Arrows with solid lines denote generative process; arrows with dashed lines denote inference; hollow arrows denote deterministic transformation leading to implicit distributions; and blue arrows denote adversarial objectives.


You can view a deep neural network as a graphical model, but here, the CPDs are not probabilistic but are deterministic. Consider for example that the input to a neuron is $\vec{x}$ and the output of the neuron is y. In the CPD for this neuron we have, $p(\vec{x},y)=1$, and $p(\vec{x},\hat{y})=0$ for $\hat{y}\neq y$. Refer to the section 10.2.3 of Deep Learning Book for more details.

  • $\begingroup$ Are RNNs fully-observed graphical models (model observations directly), or latent-variable models (introduce unobserved r.v. for every observation)? RNNs have hidden states but the Deep Learning Book states that we ignore the hidden units "by marginalizing them out of the model". And how about DNNs in general? $\endgroup$ – Christabella Irwanto Feb 6 '20 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.