In machine learning, if I understand it correctly, a generative classifier is one where we directly model the joint distribution $p(x,y|\theta)$, while in a discriminative classifier we model $p(y|x, \theta)$.

Firstly, I don't understand why they are called that way. Secondly, I don't understand what makes these models different, since we can always use the product rule to get one from the other.

  • $\begingroup$ you are getting confused with your notation - in one case $\theta$ is parameters to model $y|x$ whereas in the other case it is the parameters to model the joint distribution of x and y. $\endgroup$
    – seanv507
    Oct 9 '18 at 17:47
  • $\begingroup$ @seanv507, this still doesnt resolve my confusion $\endgroup$
    – user56834
    Oct 9 '18 at 18:04

A generative model is so called because it tries to learn the probability distribution that generated the data. For example, the Gaussian mixture model will try to learn the parameters of the Gaussian mixture that best fits the data.

A discriminative model is so called because it tries to learn which values $x$ will map to $y$, so it tries to discriminate among the inputs. Neural networks are an example.

From Wikipedia (https://en.wikipedia.org/wiki/Generative_model)

Regardless of precise definition, the terminology is because a generative model can be used to "generate" random instances [...] while a discriminative model [...] can be used to "discriminate" the value of the target variable Y, given an observation x

It is true that a generative model can be used to compute the conditional probability, but the other way around is not true.

So, why do we use discriminative models?

A generative model is typically unsupervised, similarly to clustering. A discriminative model is given a more precise task, just try to predict $y$ given $x$, so it's typically supervised. If prediction is the only thing you care about (as you do in classification tasks), discriminative models will try to solve that problem more directly and might perform better.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.