27
$\begingroup$

What are the differences between generative and discriminative (discriminant) models (in the context of Bayesian learning and inference)?

and what it is concerned with prediction, decision theory or unsupervised learning?

$\endgroup$
3
  • $\begingroup$ Sorry, I don't understand what you mean by the second sentence. Would you try and reword it? $\endgroup$ Nov 19, 2010 at 10:06
  • $\begingroup$ ohu, i'm just joined the world of statistic and machine learning, sorry i didin't find out how to link unsupervised learning with decision theory. but i'm still studing! $\endgroup$
    – nkint
    Nov 19, 2010 at 11:23
  • 2
    $\begingroup$ I'm just confused how it fits in with the question. For example, the words "prediction", "decision theory" or "unsupervised" don't appear in the accepted answer $\endgroup$ Nov 19, 2010 at 13:43

2 Answers 2

38
$\begingroup$

Both are used in supervised learning where you want to learn a rule that maps input x to output y, given a number of training examples of the form $\{(x_i,y_i)\}$. A generative model (e.g., naive Bayes) explicitly models the joint probability distribution $p(x,y)$ and then uses the Bayes rule to compute $p(y|x)$. On the other hand, a discriminative model (e.g., logistic regression) directly models $p(y|x)$.

Some people argue that the discriminative model is better in the sense that it directly models the quantity you care about $(y)$, so you don't have to spend your modeling efforts on the input x (you need to compute $p(x|y)$ as well in a generative model). However, the generative model has its own advantages such as the capability of dealing with missing data, etc. For some comparison, you can take a look at this paper: On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data). In fact, there exists hybird models too that try to bring in the best of both worlds. See this paper for an example: Principled hybrids of generative and discriminative models

$\endgroup$
2
  • 2
    $\begingroup$ Nice answer. With respect to comparing canonical examples of discriminative vs generative classifiers (logistic regression and Gaussian naive Bayes respectively), I found this book chapter to be very accessible than the Ng: cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf $\endgroup$ Nov 19, 2010 at 14:25
  • 1
    $\begingroup$ There are many gaps in this explanation. For example, Is Bayesian logistic regression considered generative or discriminative by this definition? $\endgroup$
    – Digio
    Sep 11, 2020 at 7:15
4
$\begingroup$

One addition to the above answer:

Since discriminant cares P(Y|X) only, while generative cares P(X,Y) and P(X) at the same time, in order to predict P(Y|X) well, the generative model has less degree of freedom in the model compared to discriminant model. So generative model is more robust, less prone to overfitting while discriminant is the other way around.

That explains the above answer

There can be cases when one model is better than the other (e.g., discriminative models usually tend to do better if you have lots of data; generative models may be better if you have some extra unlabeled data).

$\endgroup$
3
  • 3
    $\begingroup$ Can you explain what you're saying about the fact that generative models have less degrees of freedom? Proof? Links? Thanks $\endgroup$
    – Patrick
    Mar 26, 2019 at 22:25
  • 1
    $\begingroup$ @Patrick "Your Classifier is Secretly an Energy Based Model..." (Grathwol et al., 2019) shows a concrete example of this: cross entropy loss is invariant to shifting logits, and they remove this degree of freedom to convert discriminative models into energy-based generative models. But ArtificiallyIntelligence would perhaps have a better source. $\endgroup$ Dec 16, 2019 at 19:49
  • $\begingroup$ I don't think the single degree of freedom related to normalisation can have a noticeable effect on the capacity of a model. In any case, the pretentiously titled paper @ChristabellaIrwanto kindly cites does not seem to say anything on this topic that cannot be found in older standard texts. $\endgroup$ Jul 19 at 0:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.