Why generative models are better at detecting outliers?

Question

I've read somewhere that generative models are better than discriminative ones to detect outliers in our dataset—why is that true? I think its somehow related to decision boundaries and equiprobability curves, but can't get the entire intuition. Thanks!

Answer 1

Let $x$ be your data and $y$ be some sort of label or decision variable that this data can be mapped to. Discriminative models learn (some approximation of) $p(y|x)$. Generative models, depending on who you ask, learn $p(x)$, $p(x|y)$ or $p(x,y)$. These three are of course related. We can get the joint from the conditional as $p(x,y)=p(x|y)p(y)$, and we can get the marginal from the joint as $p(x)=\int p(x,y) dy$. In any case, the point is that they learn something about the probability of the data.

Now let's think about what an outlier is. An outlier is an observation in your data that is unlikely. That is, $p(x)$ is small (or $p(x|y)$ is small, if you are able to condition on an observed value of $y$). So you need some way of calculating $p(x)$ or $p(x|y)$. A generative model allows you to do that, through the math I described above. In words, a generative model was trained to accurately gauge the probability of your training data (assigning higher probabilities to more common values), and so given a new piece of data a generative model can tell you how probable that value is (under the model).

For a discriminative model, this is much harder, as we have just learned $p(y|x)$. How are you going to get from that to $p(x)$ or $p(x|y)$? To get the latter, you need: $$ p(x|y) = \frac{p(y|x)p(x)}{p(y)} $$ (This is basically Bayes' rule in reverse.) Note that this depends on $p(x)$, which you don't know. Rearranging the same formula, we find: $$ p(x) = \frac{p(y|x)}{p(x|y)p(y)}=\frac{p(y|x)}{p(x,y)} $$ Which again isn't helpful as it requires us to fill in terms on the r.h.s. that we don't know because they're not part of the model. In other words, because you never trained your discriminative model to tell you the probabilities of your data (only the probabilities of the labels given the data), you can't get it to do so at test time either.

For a generative model, you plug in either $x$ or the combination of $x$ and $y$, and the resulting probability, if it's small, will tell you that that particular observation is an outlier.

Answer 2

I would say that you cant really define an outlier is unless you know what sort of data you expect to see. An outlier is an unusual observation. A generative model is essentially a way of specifying the kind of data you expect to see, so if it would be generated with a very low probability by your generative model, it is by definition an outlier.