# Variational Autoencoders and Probabilistic Graphical Models

I am just getting started with the theory on variational autoencoders (VAE) in machine learning and I keep reading that VAEs belong to the category of Probabilistic Graphical Models (PGMs). As I understand it, this is because of the way we treat the input data x and the encoding vector z of latent variables, i.e. as a joint distribution p_θ(x,z), where θ is the vector of network parameters.\

However, I was wondering whether there was a way to exploit the very useful property of PGMs, namely to infer a graph structure (whether directed or undirected) representing correlations (or causal relationships in the case of DAGs) among variables in the input data. Is this possible once the latent variable vector is selected in the encoding process and would the encoder network in the VAE help in the selection of the graph structure?

• Directed acyclic graphs do not come prepackaged with a causal semantics, nor causal formalism. Only causal directed acyclic graphs do. Commented Aug 28, 2021 at 17:44
• And the standard initial setup for probabilistic graphical models is to postulate a graph structure then do parameter estimation and inference. The problem of inferring the structure of the graph itself, as a model selection problem is distinct. And given that variational autoencoders already explicitly assume a graph structure, it’s difficult to see how one proceeds to infer that which is already specified. Commented Aug 28, 2021 at 17:50

## 1 Answer

You may want to look at DAG-GNN where they essentially train a Variational autoencoder such that one layer in the encoder represents the structure $$A$$ of a Bayesian Network to be learned

One of the key challenges there is to require the learned structure to have certain graph properties, like acyclicity in the case of a DAG. In case of the DAG-GNN, the acyclicity requirement puts a non-trivial constraint on the layer weight which results in a much more involved train method.

• Thanks, I had a look at the paper and it is indeed very interesting. In the meantime I also found a reference to GraN-DAG, another algorithm used for structure learning of DAGs which uses standard multilayer neural network to learn the graph structure. Commented Aug 30, 2021 at 11:36
• However, in both papers, they use the Structural Hamming Distance or the Structural Interventional Distance to measure the performance of their algorithms. This requires a "gold standard DAG" to compare the output against, which we do not necessarily have in many applications. Since you seem knowledgeable in this field, do you think there are any issues in using other metrics like the log-likelihood to assess the fit of the model? Commented Aug 30, 2021 at 11:36
• As you said, in many applications you cannot compare performance to a "gold standard" graph (as you want to find such a thing). Moreover, you probably also do not have identifiable graphical models in applications (as required by the GraN-DAG paper). I think that relying solely on the log-likelihood to evaluate the model fit depends on what exactly your goals are. If you want to learn a graphical model in many dimensions with a method like that (and without interventional data), you have not much choices than log-likelihood or BIC-like scores. Commented Sep 1, 2021 at 20:10