Do variational approximations capture the flow of influence or "conditional independence" relationships in graphical models? Probabilistic Graphical Models (PGMs) are used to model all sorts of complex decision processes, such as medical diagnoses or robot positions, etc. In common machine learning textbooks, like Christopher Bishops book on pattern recognition or Koller's text, the idea of the "flow of influence" or conditional independence seems to be a big feature/benefit of these methods. In other words, these models can make better decisions when dealing with data where a mixture of generative processes may exist.
I am a bit confused about the inference methods for PGMs versus the ability of these inference methods to capture the conditional independence relationships between variables. In both the Bishop, Koller, and Murphy texts, the decription of Bayesian networks (directed) and Markov random fields, is immediately followed by the discussion about "exact" inference and Belief Propagation. Then variational methods are kept for a subsequent chapter.
I can see how belief propagation or message passing methods execute inference by following the pattern of conditional independence relationships in the network. But my question is, do the approximate inference methods, like maximizing the ELBO, evidence lower bound, also capture this pattern of conditional independence relationships? Also, under what conditions do variational inference methods fail to capture the flow of influence in the original network. My assumption is the variational methods should factor in the conditional independence structure of the network, but there should be issues with the complexity of the underlying distribution, or with the mis-specification of the candidate factorized distribution for VI that could get in the way of accurate representation.
Part of my confusion also stems from the fact that many of the popular "Probabilistic Programming" packages, like Pyro, or Tensorflow Probability, or even PyMC3 don't support any belief propagation/message passing methods, and their tutorials never really discuss conditional independence and the flow of influence. So I was just hoping to find an explicit answer to this question.
 A: The optimization process casts by variational inference consists in fitting a function to an another (a probability distribution to another (simpler) probability distribution, we call the latter the variational distribution) under a criteria to maximize (ELBO most commonly). I think it is the simplest way to think about the problem.
The most common approach to variational inference is mean field variational inference which uses a variational distribution with fully independent random variables (no structure at all, also called fully factorized). This approach does not try to replicate any conditional independence in the variational distribution. Thus we do not learn conditional independence relations. This leads to a direct inference procedure but might fit the original complex distribution quite badly.
It is possible to model your variational distribution with conditional independence between its random variables. It is called structured variational inference (short review Section 5.3 here). This structured variational inference approach can be seen as a way to try to fit the original distribution by a more refined family of distribution. If your structured variational distribution is well crafted you can fit much better your target distribution than in a mean field approach (but with a possibly more complex inference in the variational distribution this time). If some conditional independence relations are learnt in the process it is because the family of function you allow yourself to use somehow already encodes that relations (to the best of my knowledge :) ).
