# Why use KL-Divergence as loss over MLE?

I have came across this statement several time now

Maximizing likelihood is equivalent to minimizing KL-Divergence

I would like to know in applications such as VAE, why use KL- divergence then over MLE? In which applications would you choose one over the other? And any specific reason for it given both are equivalent?

• What is the real question here? The statement has been proven in a linked question. And it is helpful in the sense, that (at least to my limited understanding of VAE) it can actually be computed, at least for reasonable choices for the prior distribution (e.g. a Gaussian). I wouldn't really know how or where to start to find a proper expression for the Likelihood. Mar 5, 2021 at 21:29
• I think then I am missing something fundamental in my understanding. when i started typing my explanation I think I realized whats the problem and what you mean. When we usually use NLL we use it as a loss function, where we have the ideal output to model our distribution on, hence NLL. But in mapping to latent variable space, we dont have the required output ideal latent distribution (its unknown) therefore we cant fit it to any distribution using NLL. is that right? Mar 6, 2021 at 4:46
• What i had in mind was something on the lines of assuming that the encoded latent variable is normal, and then using that to fit a prior normal N(0,1) to it by taking its loss using NLL and adding it to the final decoder NLL loss. Mar 6, 2021 at 4:54
• What does NLL stand for? Mar 6, 2021 at 10:09
• NLL negative log likelihood. its for maximum likelihood estimation. stats.stackexchange.com/questions/141087/… Mar 6, 2021 at 13:04

Although maximizing likelihood is equivalent to minimizing KL divergence from the data distribution to the model, this doesn't mean that every application of KL divergence is maximum likelihood, because often, the two things you're measuring divergence between are not data and model.

In particular, VAEs are trained by maximizing something which is a lower bound on the likelihood, so in a sense they are really just trained by MLE. It happens that the lower bound has as one of its terms the KL divergence between the variational distribution $$q(z|X)$$ and the latent prior $$p(z)$$. But since these aren't data and model, it doesn't make sense to think of the KL term as MLE.

• Hi, I think it makes of more sense now. can u also give somewhat detailed reading on the topic if possible? textbook perhaps? Mar 10, 2021 at 4:22
• @shimao I'm still relatively new to VAE. My understanding in rough terms is opposite of what you've written. In principle there is no model, so you can't compute a likelihood. So you use the KL divergence, since this can be computed to get some kind of performance metric for the training. So the "[VAEs] are really just trained by MLE" should be "only trained by KL"!? Mar 10, 2021 at 14:29
• @cherub Not sure what you mean by "no model". The VAE model is $p(x) = \int p(x|z)p(z) dz$, there is a likelihood, it's just intractable. You optimize the ELBO, which is a lower bound on the likelihood. Is it fair to say that maximizing a lower bound on the likelihood is "maximum likelihood"? Well, I think it's close enough, but it seems like just a matter of semantics. Mar 10, 2021 at 14:34
• @shimao OK, thanks for the clarification. ELBO is the "evidence lower bound", a term which I hadn't come across before. I need a bit more thought to take this in; but it's likely reduced to semantics. I'm approaching the whole topic from the "other" side, in the sense that there is always a model to start with; hence my strict interpretation of max likelhood. It looks like this is a very good point to "explain" VAEs, that you introduce this formulation. Mar 10, 2021 at 15:45