1
$\begingroup$

I am just getting started with the theory on variational autoencoders (VAE) in machine learning and I keep reading that VAEs belong to the category of Probabilistic Graphical Models (PGMs). As I understand it, this is because of the way we treat the input data x and the encoding vector z of latent variables, i.e. as a joint distribution p_θ(x,z), where θ is the vector of network parameters.\

However, I was wondering whether there was a way to exploit the very useful property of PGMs, namely to infer a graph structure (whether directed or undirected) representing correlations (or causal relationships in the case of DAGs) among variables in the input data. Is this possible once the latent variable vector is selected in the encoding process and would the encoder network in the VAE help in the selection of the graph structure?

$\endgroup$
2
  • 2
    $\begingroup$ Directed acyclic graphs do not come prepackaged with a causal semantics, nor causal formalism. Only causal directed acyclic graphs do. $\endgroup$
    – microhaus
    Aug 28, 2021 at 17:44
  • $\begingroup$ And the standard initial setup for probabilistic graphical models is to postulate a graph structure then do parameter estimation and inference. The problem of inferring the structure of the graph itself, as a model selection problem is distinct. And given that variational autoencoders already explicitly assume a graph structure, it’s difficult to see how one proceeds to infer that which is already specified. $\endgroup$
    – microhaus
    Aug 28, 2021 at 17:50

1 Answer 1

1
$\begingroup$

You may want to look at DAG-GNN where they essentially train a Variational autoencoder such that one layer in the encoder represents the structure $A$ of a Bayesian Network to be learned

enter image description here

One of the key challenges there is to require the learned structure to have certain graph properties, like acyclicity in the case of a DAG. In case of the DAG-GNN, the acyclicity requirement puts a non-trivial constraint on the layer weight which results in a much more involved train method.

$\endgroup$
3
  • 1
    $\begingroup$ Thanks, I had a look at the paper and it is indeed very interesting. In the meantime I also found a reference to GraN-DAG, another algorithm used for structure learning of DAGs which uses standard multilayer neural network to learn the graph structure. $\endgroup$ Aug 30, 2021 at 11:36
  • 1
    $\begingroup$ However, in both papers, they use the Structural Hamming Distance or the Structural Interventional Distance to measure the performance of their algorithms. This requires a "gold standard DAG" to compare the output against, which we do not necessarily have in many applications. Since you seem knowledgeable in this field, do you think there are any issues in using other metrics like the log-likelihood to assess the fit of the model? $\endgroup$ Aug 30, 2021 at 11:36
  • $\begingroup$ As you said, in many applications you cannot compare performance to a "gold standard" graph (as you want to find such a thing). Moreover, you probably also do not have identifiable graphical models in applications (as required by the GraN-DAG paper). I think that relying solely on the log-likelihood to evaluate the model fit depends on what exactly your goals are. If you want to learn a graphical model in many dimensions with a method like that (and without interventional data), you have not much choices than log-likelihood or BIC-like scores. $\endgroup$ Sep 1, 2021 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.