Generative vs discriminative models (in Bayesian context)

Question

What are the differences between generative and discriminative (discriminant) models (in the context of Bayesian learning and inference)?

and what it is concerned with prediction, decision theory or unsupervised learning?

Answer 1

Both are used in supervised learning where you want to learn a rule that maps input x to output y, given a number of training examples of the form $\{(x_i,y_i)\}$. A generative model (e.g., naive Bayes) explicitly models the joint probability distribution $p(x,y)$ and then uses the Bayes rule to compute $p(y|x)$. On the other hand, a discriminative model (e.g., logistic regression) directly models $p(y|x)$.

Some people argue that the discriminative model is better in the sense that it directly models the quantity you care about $(y)$, so you don't have to spend your modeling efforts on the input x (you need to compute $p(x|y)$ as well in a generative model). However, the generative model has its own advantages such as the capability of dealing with missing data, etc. For some comparison, you can take a look at this paper: On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data). In fact, there exists hybird models too that try to bring in the best of both worlds. See this paper for an example: Principled hybrids of generative and discriminative models

Answer 2

One addition to the above answer:

Since discriminant cares P(Y|X) only, while generative cares P(X,Y) and P(X) at the same time, in order to predict P(Y|X) well, the generative model has less degree of freedom in the model compared to discriminant model. So generative model is more robust, less prone to overfitting while discriminant is the other way around.

That explains the above answer

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data).