1
$\begingroup$

I feel that most generative models happen to be DGM(directed graphical model), and most discriminative models are UGM(undirected graphical model). Is there any correlation between these concepts?

  • generative model and DGM, e.g. Navie Bayes/Mixture Gaussian/HMM
  • discriminative model and UGM, e.g. Logistic regression/CRFs
  • generative model and UGM: ??? Ising Model ($p(x,y)=\frac{exp(-E(x,y))}{Z}$)
  • discriminative and DGM: any examples?

In generative model(or DGM), we model the joint probability $P(X,Y|\theta)$($\theta$ is hyperparameter, $X$ is hidden variable). The objective is to maximize the marginals $P(Y|\theta)$ w.r.t $\theta$. Because we specify the joint distribution, we can make inference about $X$.

For discriminative model(or UGM), we specify $P(Y|X,\theta)$ and try to maximize it directly. e.g. Logistic regression, $logit[P(Y_n=1|x_n)]=\beta x_n$

$\endgroup$
  • $\begingroup$ I found the example of discriminative DGM, maximum-entropy Markov model (MEMM). My feeling is that most DGM are generative. $\endgroup$ – H.Yuanchen Apr 18 '19 at 15:52
0
$\begingroup$

Directed Graphical Models, or Bayesian Networks, model the joint distribution with a specific factorization. Thus, when using DGM, you always consider the step of writing the joint law $p(X,Y)$ from which you can deduce $p(X)$ and $p(Y|X)$ and what you get is a generative model (with $X$ being the hidden variable and $Y$ the observed variable).

On the other hand, when using an Undirected Graphical Model, you must consider writing potential functions which a are much less intuitive and harder to use than conditional probabilities used in DGM. That may be why you are feeling this tendency of UGM to be discriminative : energy functions are hard to write, "don't waste ressources" to model the full $p(X,Y)$ if $p(X|Y)$ is all that interests you. (Quote from Machine Learning, Chapter 19, Kevin P; Murphy)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.