I understand how KL divergence provides us with a measure of how one probability distribution is different from a second, reference probability distribution. But why is it particularly used (instead of cross-entropy) in generative networks such as Variational Autoencoders (VAEs)? As much as I understand, minimizing either Cross-Entropy or KL-divergence is equivalent. So, I struggle to understand why KL divergence is the preferred loss function in VAEs.

NB: I originally asked this question in Data Science SE and referred to forward it here.

  • 2
    $\begingroup$ The last 2 sections on KLD and CE might help you understand for VAE medium.com/swlh/cross-entropy-and-kl-divergence-522d9f71bd3d $\endgroup$
    – develarist
    Sep 25, 2020 at 8:14
  • $\begingroup$ related: datascience.stackexchange.com/questions/82152/… $\endgroup$
    – denfromufa
    Aug 3, 2022 at 15:45
  • $\begingroup$ Please don't cross-post. If you feel that another site would be a better fit for the question, flag it for migration. $\endgroup$
    – Sycorax
    Aug 3, 2022 at 16:38
  • 2
    $\begingroup$ KL is not the same as cross entropy; KL is negative self entropy + cross entropy. So minimizing KL has an additional regularizing effect (via the entropy maximization) absent in CE minimization. $\endgroup$ Aug 3, 2022 at 17:35
  • 1
    $\begingroup$ @JohnMadden That seems like the start of a complete answer to the question. Perhaps you could expand it? $\endgroup$
    – Sycorax
    Aug 3, 2022 at 17:36


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.