Is the only difference between conditional generative models and discriminative models the complexity of the modeled distribution?

Question

One common way to define discriminative models is that they model $P(Y|X)$, where $Y$ is the label, and $X$ is the observed variables. Conditional generative models do something quite similar, but the modeled distribution of $Y$ tends to be more complex -- for example, $Y$ might be a distribution of images, where the conditioning variable $X$ might be the image class. Usually it is quite intuitive whether a model is discriminative, or conditional generative, but in some cases it seems quite unclear, so I wonder: is the distinction between the two purely arbitrary?

Here is a list of models of $P(Y|X)$ for different $Y$ and $X$, listed in order of increasing "generativeness"

1. A semantic segmentation model (a convolutional network) outputting a distribution over classes for each pixel in an image.

2. A convolutional network which regresses dense monocular depth -- outputting a normal distribution over depths for each pixel in an image.

3. Same as 4, but with no conditioning noise in the generator.

4. A GAN which outputs dense monocular depth from RGB input, using both the typical GAN loss and also a regression loss.

5. A cycleGAN model which lets you sample from the distribution $Y|X$, where $X$ is an image, and $Y$ is a rendition of that image in the style of a Monet painting.

6. A conditional GAN model which generates a photorealistic image of any class of animal (the animal label being the conditioning variable).

I think almost everyone agrees 1 and 6 are discriminative and conditional generative respectively. The same for 2 and 5 respectively, but the justification is less clear. Finally for 3 and 4, it's quite unclear to me how they should be classified. 3 models a far less complex distribution over dense depth maps because of the lack of conditioning noise, and leans towards the discriminative side, and vice versa for 4.

So: 1: Is there a clean distinction between discriminative and conditional generative models? 2: Is it just some arbitrary function of how complex the modeled distribution is? 3: If there is a good definition, how would it classify the above models?

Answer 1

In machine learning, the terms "generative" and "discriminative" are sometimes conflated with "unsupervised" and "supervised". Someone only reading deep learning papers may also come to believe that generative models "generate" images, text, or audio and discriminative models assign labels. But this is not how these terms were defined and used before GANs and VAEs came along.

A generative model assigns a joint probability distribution to all variables involved, even if we ultimately only care about a conditional or marginal distribution. Classical examples of generative models include the naive Bayes classifier and latent Dirichlet allocation. Naive Bayes is supervised and is usually applied to very simple data, so data complexity does not matter at all. And neither of these models was designed to generate inputs. In fact, their conditional independence assumptions makes them do a poor job of it.

A discriminative or conditional model assigns a conditional probability to one set of variables given another set of variables. Discriminative models are sometimes trained in an unsupervised manner, see discriminative clustering.

More general terms encompassing both are "probabilistic model" or "statistical model", which can refer to any collection of probability distributions. We use "generative" and "discriminative" to quickly communicate some general properties of a probabilistic model. Other specifiers include "parametric", "nonparametric" and "graphical".

The more recent phrase "conditional generative" suggests that we are conditioning on a variable, as in discriminative modeling, but that we are also modeling distributions which are not actually of interest at test time, as in generative modeling. E.g., when training a conditional GAN to generate images $x$ from noise $z$ and a label $c$, $p(x, z \mid c)$, we probably only care about $p(x \mid c)$ at test time. But to be consistent with the definitions of "generative" and "discriminative", I propose the complexity or type of data should not matter when deciding whether something is "conditionally generative" or not.

Using these definitions, we would assign labels as follows:

1. discriminative
2. discriminative, since the type of data or distribution does not matter
3. discriminative, since you are only conditionally modeling variables of interest
4. conditional generative, since you model auxiliary variables; the loss you use to train a model does not matter
5. discriminative, the complexity or type of data, or the way you train your model(s) does not matter
6. conditional generative

I'll point to a note by Minka (2005) and a paper by Bishop & Lasserre (2007) which provide more precise if narrower definitions and a unified view of generative and discriminative models which may clear things up further.

Finally, note that the term "model" is overloaded and can refer both to a family of distributions and to an instance of such a family. For example, the "trained model" $p_{\theta}$ with particular parameter values is an instance of a parametric model, $\{ p_\theta : \theta \in \mathbb{R}^N \}$. To make things even more confusing, machine learning terminology conflates models with fitting procedures. Note that VAEs and GANs (and nonlinear ICA and some others) are referring to essentially the same model (if you assume very small $\sigma$),

$$\mathcal{N}(z; 0, I)\mathcal{N}(x; f_\theta(z); \sigma^2 I),$$

yet we often talk about them as different "models" because they are optimized differently.