4
$\begingroup$

I have came across this statement several time now

Maximizing likelihood is equivalent to minimizing KL-Divergence

(Sources: Kullback–Leibler divergence and Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution)

I would like to know in applications such as VAE, why use KL- divergence then over MLE? In which applications would you choose one over the other? And any specific reason for it given both are equivalent?

$\endgroup$
7
  • $\begingroup$ What is the real question here? The statement has been proven in a linked question. And it is helpful in the sense, that (at least to my limited understanding of VAE) it can actually be computed, at least for reasonable choices for the prior distribution (e.g. a Gaussian). I wouldn't really know how or where to start to find a proper expression for the Likelihood. $\endgroup$
    – cherub
    Commented Mar 5, 2021 at 21:29
  • $\begingroup$ I think then I am missing something fundamental in my understanding. when i started typing my explanation I think I realized whats the problem and what you mean. When we usually use NLL we use it as a loss function, where we have the ideal output to model our distribution on, hence NLL. But in mapping to latent variable space, we dont have the required output ideal latent distribution (its unknown) therefore we cant fit it to any distribution using NLL. is that right? $\endgroup$
    – ipcamit
    Commented Mar 6, 2021 at 4:46
  • $\begingroup$ What i had in mind was something on the lines of assuming that the encoded latent variable is normal, and then using that to fit a prior normal N(0,1) to it by taking its loss using NLL and adding it to the final decoder NLL loss. $\endgroup$
    – ipcamit
    Commented Mar 6, 2021 at 4:54
  • 1
    $\begingroup$ What does NLL stand for? $\endgroup$ Commented Mar 6, 2021 at 10:09
  • $\begingroup$ NLL negative log likelihood. its for maximum likelihood estimation. stats.stackexchange.com/questions/141087/… $\endgroup$
    – ipcamit
    Commented Mar 6, 2021 at 13:04

1 Answer 1

3
+50
$\begingroup$

Although maximizing likelihood is equivalent to minimizing KL divergence from the data distribution to the model, this doesn't mean that every application of KL divergence is maximum likelihood, because often, the two things you're measuring divergence between are not data and model.

In particular, VAEs are trained by maximizing something which is a lower bound on the likelihood, so in a sense they are really just trained by MLE. It happens that the lower bound has as one of its terms the KL divergence between the variational distribution $q(z|X)$ and the latent prior $p(z)$. But since these aren't data and model, it doesn't make sense to think of the KL term as MLE.

$\endgroup$
4
  • $\begingroup$ Hi, I think it makes of more sense now. can u also give somewhat detailed reading on the topic if possible? textbook perhaps? $\endgroup$
    – ipcamit
    Commented Mar 10, 2021 at 4:22
  • $\begingroup$ @shimao I'm still relatively new to VAE. My understanding in rough terms is opposite of what you've written. In principle there is no model, so you can't compute a likelihood. So you use the KL divergence, since this can be computed to get some kind of performance metric for the training. So the "[VAEs] are really just trained by MLE" should be "only trained by KL"!? $\endgroup$
    – cherub
    Commented Mar 10, 2021 at 14:29
  • 2
    $\begingroup$ @cherub Not sure what you mean by "no model". The VAE model is $p(x) = \int p(x|z)p(z) dz$, there is a likelihood, it's just intractable. You optimize the ELBO, which is a lower bound on the likelihood. Is it fair to say that maximizing a lower bound on the likelihood is "maximum likelihood"? Well, I think it's close enough, but it seems like just a matter of semantics. $\endgroup$
    – shimao
    Commented Mar 10, 2021 at 14:34
  • $\begingroup$ @shimao OK, thanks for the clarification. ELBO is the "evidence lower bound", a term which I hadn't come across before. I need a bit more thought to take this in; but it's likely reduced to semantics. I'm approaching the whole topic from the "other" side, in the sense that there is always a model to start with; hence my strict interpretation of max likelhood. It looks like this is a very good point to "explain" VAEs, that you introduce this formulation. $\endgroup$
    – cherub
    Commented Mar 10, 2021 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.