What is the relationship between discriminative and generative and directed and undirected graphical models?

Question

I feel that most generative models happen to be DGM(directed graphical model), and most discriminative models are UGM(undirected graphical model). Is there any correlation between these concepts?

• generative model and DGM, e.g. Navie Bayes/Mixture Gaussian/HMM
• discriminative model and UGM, e.g. Logistic regression/CRFs
• generative model and UGM: ??? Ising Model ($p(x,y)=\frac{exp(-E(x,y))}{Z}$)
• discriminative and DGM: any examples?

In generative model(or DGM), we model the joint probability $P(X,Y|\theta)$($\theta$ is hyperparameter, $X$ is hidden variable). The objective is to maximize the marginals $P(Y|\theta)$ w.r.t $\theta$. Because we specify the joint distribution, we can make inference about $X$.

For discriminative model(or UGM), we specify $P(Y|X,\theta)$ and try to maximize it directly. e.g. Logistic regression, $logit[P(Y_n=1|x_n)]=\beta x_n$

Answer 1

Directed Graphical Models, or Bayesian Networks, model the joint distribution with a specific factorization. Thus, when using DGM, you always consider the step of writing the joint law $p(X,Y)$ from which you can deduce $p(X)$ and $p(Y|X)$ and what you get is a generative model (with $X$ being the hidden variable and $Y$ the observed variable).

On the other hand, when using an Undirected Graphical Model, you must consider writing potential functions which a are much less intuitive and harder to use than conditional probabilities used in DGM. That may be why you are feeling this tendency of UGM to be discriminative : energy functions are hard to write, "don't waste ressources" to model the full $p(X,Y)$ if $p(X|Y)$ is all that interests you. (Quote from Machine Learning, Chapter 19, Kevin P; Murphy)