22
$\begingroup$

What are the differences between generative and discriminative (discriminant) models (in the context of Bayesian learning and inference)?

and what it is concerned with prediction, decision theory or unsupervised learning?

$\endgroup$
  • $\begingroup$ Sorry, I don't understand what you mean by the second sentence. Would you try and reword it? $\endgroup$ – csgillespie Nov 19 '10 at 10:06
  • $\begingroup$ ohu, i'm just joined the world of statistic and machine learning, sorry i didin't find out how to link unsupervised learning with decision theory. but i'm still studing! $\endgroup$ – nkint Nov 19 '10 at 11:23
  • 1
    $\begingroup$ I'm just confused how it fits in with the question. For example, the words "prediction", "decision theory" or "unsupervised" don't appear in the accepted answer $\endgroup$ – csgillespie Nov 19 '10 at 13:43
35
$\begingroup$

Both are used in supervised learning where you want to learn a rule that maps input x to output y, given a number of training examples of the form $\{(x_i,y_i)\}$. A generative model (e.g., naive Bayes) explicitly models the joint probability distribution $p(x,y)$ and then uses the Bayes rule to compute $p(y|x)$. On the other hand, a discriminative model (e.g., logistic regression) directly models $p(y|x)$.

Some people argue that the discriminative model is better in the sense that it directly models the quantity you care about $(y)$, so you don't have to spend your modeling efforts on the input x (you need to compute $p(x|y)$ as well in a generative model). However, the generative model has its own advantages such as the capability of dealing with missing data, etc. For some comparison, you can take a look at this paper: On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data). In fact, there exists hybird models too that try to bring in the best of both worlds. See this paper for an example: Principled hybrids of generative and discriminative models

$\endgroup$
  • 1
    $\begingroup$ Nice answer. With respect to comparing canonical examples of discriminative vs generative classifiers (logistic regression and Gaussian naive Bayes respectively), I found this book chapter to be very accessible than the Ng: cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf $\endgroup$ – Josh Hemann Nov 19 '10 at 14:25
3
$\begingroup$

One addition to the above answer:

Since discriminant cares P(Y|X) only, while generative cares P(X,Y) and P(X) at the same time, in order to predict P(Y|X) well, the generative model has less degree of freedom in the model compared to discriminant model. So generative model is more robust, less prone to overfitting while discriminant is the other way around.

That explains the above answer

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data).

$\endgroup$
  • 2
    $\begingroup$ Can you explain what you're saying about the fact that generative models have less degrees of freedom? Proof? Links? Thanks $\endgroup$ – Patrick Mar 26 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.