Difference / Relationship of Generative Models / Variational Bayesian Inference

Question

I feel a bit confused trying to merge and unify understandings of generative models and variational bayesian inference methods. Initially, I believed them to be the same thing, namely learning full posterior probabiltiy distributions, s.t. one can sample from them, and generate data samples.

Discriminative methods, afaik, learn p(y|x), i.e. a decision boundary for making a prediction. Generative models, on the other hand learn p(y, x) = p(y|x) * p(x), and we can create samples from that. Now, as far as I understand, VI is a method of approximating posterior distributions over latent variables, i.e. p(z|x) (but, from my understanding, whether it is a posterior of some z or y does not matter). Which now again looks more similar to discriminate models though ...

Is my confusion a bit clear? It would be great if someone could help me with that!

Answer 1

Your confusion might come from the fact that the principle of VI is to form an approximation of a posterior distribution in which inference is easy. This means we are often able to sample, generate data from this posterior (generate samples of the hidden variables conditioned on the observed variables). This is not because we are able to sample data that we have a generative model in the common sense you mention (joint distribution observed and hidden variables).

Thus, VI can be used in discriminative methods, as a way to simplify inference in a complex model, for example in Conditional Random Fields.