6
$\begingroup$

I was reading about the difference between discriminative and generative models, and I read that Discriminative models learn only the boundary between classes hence they are not able to to create new datapoints. The article states, if we use a generative algorithm, for instance naive bayes, we can create new data points from class “i” basically by choosing features that maximize P(X|Y=class i ). However; the point I did not understand is that could not we also do the same creating in a discriminative model.

Take logistic regression for instance, cannot we create a vector x that maximizes P(Y=i|X=x) ? Would not this x vector would be our new sample , namely did not we generated a new sample ?

$\endgroup$
5
  • 3
    $\begingroup$ Does this help stats.stackexchange.com/questions/12421/… ? TL;DR you would need to know P(X) to get P(Y|X) P(X) = P(Y, X). $\endgroup$
    – Tim
    Jun 22, 2022 at 8:50
  • $\begingroup$ I read this one. However I think it does not completely answers my question of why cannot we create new datapoints by basically chosing a vector x that maximizes the probability. $\endgroup$
    – levitatmas
    Jun 22, 2022 at 9:01
  • $\begingroup$ You can, but this doesn't make it a generative model. $\endgroup$
    – Tim
    Jun 22, 2022 at 9:08
  • $\begingroup$ What makes a model generative if being able to create new datapoints does not ? I am asking cause I think there is a conceptual thing I did not understand. $\endgroup$
    – levitatmas
    Jun 22, 2022 at 9:10
  • 3
    $\begingroup$ This is how generative vs discriminative models are defined. If you sample from P(Y|X) without knowing P(X), for example by picking arbitrary values for X, or mode as you suggest, the "generated" data won't be consistent with the actual distribution of the data, hence the model does not allow you to sample from the distribution. $\endgroup$
    – Tim
    Jun 22, 2022 at 9:17

2 Answers 2

16
$\begingroup$

The fundamental difference between Generative Model and Discriminative Model is, one is learning about $ P(X,y) $ while discriminative model is learning $ P(y|X) $

According to this definition, Logistic Regression is not a generative model.

For your example "create a vector x that maximizes $ P(y=i|X=x)$ ", it was not a generative model at all. Since it still learn nothing about $ P(X,y) $

Taking a real life example, suppose we got a logistic regression model that predict whether the image is a photo of cat. (Usually this is CNN ,but the logic is the same)

The image $ X $ that maximize $ P(y= cat| X=x) $ is an image that consist of different feature of cats (e.g. tail, eyes) everywhere inside the image. Certain is maximize the probability of being classified as a cat, but you never see this image in reality. Mathematically speaking, if $ x^* =argmax(P(y= cat| X=x))$ , $ P(x^*,y) $ can still be very low.

$\endgroup$
2
  • $\begingroup$ So what you are saying is that the vector x we found might be maximizing the probability but still does not come from the distribution that we ultimately want to generate artificial samples , namely it is basically a coincidence sample that does not follow the distribution pattern we are seeking for , am I correct ? $\endgroup$
    – levitatmas
    Jun 22, 2022 at 10:22
  • $\begingroup$ @levitatmas Yes, sample maximizing the probabiltiy is different from the samepl that we want to generate. $\endgroup$
    – Bayesian
    Jun 23, 2022 at 3:22
9
$\begingroup$

To elaborate on @Bayesian's (correct) answer, consider a logistic regression model where cases of diabetes ($y$) are predicted by sugar intake ($x$).

The model learns $P(y = 1 | x) = \text{logit}^{-1}(\alpha + \beta x)$ - that is, $P(\text{Diabetes}|\text{Sugar intake})$, but since it doesn't learn the distribution of sugar intakes in the population, $P(x)$, it can't generate samples from the distribution of diabetes cases, $P(y)$.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer , I got it. As a side question, could you please explain what is meant by learning P(x) , and if it can be learned, how can one learn it. Is it by counting the number of people in whole population who have a specific sugar intake level , lets say k , so P(X=k) is obtained. And what makes this so hard is examining the whole population maybe ? $\endgroup$
    – levitatmas
    Jun 22, 2022 at 11:42
  • 3
    $\begingroup$ Yes, $P(x)$ is just information about the distribution of $x$ in the population. It's not that it's hard to obtain, it's just not part of the logistic regression model. $\endgroup$
    – Eoin
    Jun 22, 2022 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.