3
$\begingroup$

I understand how KL divergence provides us with a measure of how one probability distribution is different from a second, reference probability distribution. But why is it particularly used (instead of cross-entropy) in generative networks such as Variational Autoencoders (VAEs)? As much as I understand, minimizing either Cross-Entropy or KL-divergence is equivalent. So, I struggle to understand why KL divergence is the preferred loss function in VAEs.

NB: I originally asked this question in Data Science SE and referred to forward it here.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ The last 2 sections on KLD and CE might help you understand for VAE medium.com/swlh/cross-entropy-and-kl-divergence-522d9f71bd3d $\endgroup$
    – develarist
    Sep 25, 2020 at 8:14
  • $\begingroup$ related: datascience.stackexchange.com/questions/82152/… $\endgroup$
    – denfromufa
    Aug 3, 2022 at 15:45
  • $\begingroup$ Please don't cross-post. If you feel that another site would be a better fit for the question, flag it for migration. $\endgroup$
    – Sycorax ♦
    Aug 3, 2022 at 16:38
  • 2
    $\begingroup$ KL is not the same as cross entropy; KL is negative self entropy + cross entropy. So minimizing KL has an additional regularizing effect (via the entropy maximization) absent in CE minimization. $\endgroup$
    – John Madden
    Aug 3, 2022 at 17:35
  • 1
    $\begingroup$ @JohnMadden That seems like the start of a complete answer to the question. Perhaps you could expand it? $\endgroup$
    – Sycorax ♦
    Aug 3, 2022 at 17:36

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.