# Why use KL-Divergence as loss over MLE?

I have came across this statement several time now

Maximizing likelihood is equivalent to minimizing KL-Divergence

I would like to know in applications such as VAE, why use KL- divergence then over MLE? In which applications would you choose one over the other? And any specific reason for it given both are equivalent?

• What is the real question here? The statement has been proven in a linked question. And it is helpful in the sense, that (at least to my limited understanding of VAE) it can actually be computed, at least for reasonable choices for the prior distribution (e.g. a Gaussian). I wouldn't really know how or where to start to find a proper expression for the Likelihood. Mar 5, 2021 at 21:29
• I think then I am missing something fundamental in my understanding. when i started typing my explanation I think I realized whats the problem and what you mean. When we usually use NLL we use it as a loss function, where we have the ideal output to model our distribution on, hence NLL. But in mapping to latent variable space, we dont have the required output ideal latent distribution (its unknown) therefore we cant fit it to any distribution using NLL. is that right? Mar 6, 2021 at 4:46
• What i had in mind was something on the lines of assuming that the encoded latent variable is normal, and then using that to fit a prior normal N(0,1) to it by taking its loss using NLL and adding it to the final decoder NLL loss. Mar 6, 2021 at 4:54
• What does NLL stand for? Mar 6, 2021 at 10:09
• NLL negative log likelihood. its for maximum likelihood estimation. stats.stackexchange.com/questions/141087/… Mar 6, 2021 at 13:04

In particular, VAEs are trained by maximizing something which is a lower bound on the likelihood, so in a sense they are really just trained by MLE. It happens that the lower bound has as one of its terms the KL divergence between the variational distribution $$q(z|X)$$ and the latent prior $$p(z)$$. But since these aren't data and model, it doesn't make sense to think of the KL term as MLE.
• @cherub Not sure what you mean by "no model". The VAE model is $p(x) = \int p(x|z)p(z) dz$, there is a likelihood, it's just intractable. You optimize the ELBO, which is a lower bound on the likelihood. Is it fair to say that maximizing a lower bound on the likelihood is "maximum likelihood"? Well, I think it's close enough, but it seems like just a matter of semantics. Mar 10, 2021 at 14:34