Generative vs discriminative models (in Bayesian context)

What are the differences between generative and discriminative (discriminant) models (in the context of Bayesian learning and inference)?

and what it is concerned with prediction, decision theory or unsupervised learning?

• Sorry, I don't understand what you mean by the second sentence. Would you try and reword it? – csgillespie Nov 19 '10 at 10:06
• ohu, i'm just joined the world of statistic and machine learning, sorry i didin't find out how to link unsupervised learning with decision theory. but i'm still studing! – nkint Nov 19 '10 at 11:23
• I'm just confused how it fits in with the question. For example, the words "prediction", "decision theory" or "unsupervised" don't appear in the accepted answer – csgillespie Nov 19 '10 at 13:43

Both are used in supervised learning where you want to learn a rule that maps input x to output y, given a number of training examples of the form $\{(x_i,y_i)\}$. A generative model (e.g., naive Bayes) explicitly models the joint probability distribution $p(x,y)$ and then uses the Bayes rule to compute $p(y|x)$. On the other hand, a discriminative model (e.g., logistic regression) directly models $p(y|x)$.

Some people argue that the discriminative model is better in the sense that it directly models the quantity you care about $(y)$, so you don't have to spend your modeling efforts on the input x (you need to compute $p(x|y)$ as well in a generative model). However, the generative model has its own advantages such as the capability of dealing with missing data, etc. For some comparison, you can take a look at this paper: On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data). In fact, there exists hybird models too that try to bring in the best of both worlds. See this paper for an example: Principled hybrids of generative and discriminative models

• Nice answer. With respect to comparing canonical examples of discriminative vs generative classifiers (logistic regression and Gaussian naive Bayes respectively), I found this book chapter to be very accessible than the Ng: cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf – Josh Hemann Nov 19 '10 at 14:25