# Tag Info

Accepted

### Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

KL divergence is a natural way to measure the difference between two probability distributions. The entropy $H(p)$ of a distribution $p$ gives the minimum possible number of bits per message that ...
• 33.1k
Accepted

### What is the difference between Cross-entropy and KL divergence?

You will need some conditions to claim the equivalence between minimizing cross entropy and minimizing KL divergence. I will put your question under the context of classification problems using cross ...
• 1,848
Accepted

• 25.2k

### Is it possible to apply KL divergence between discrete and continuous distribution?

Yes, the KL divergence between continuous and discrete random variables is well defined. If $P$ and $Q$ are distributions on some space $\mathbb{X}$, then both $P$ and $Q$ have densities $f$, $g$ with ...
• 819
Accepted

### Why don't we use a symmetric cross-entropy loss?

Consider a classification context like you mentioned, where $q(y \mid x)$ is the model distribution over classes, given input $x$. $p(y \mid x)$ is the 'true' distribution, defined as a delta function ...
• 33.1k

### Interpretation of Radon-Nikodym derivative between probability measures?

First, we don't need probability measures, just $\sigma$-finiteness. So let $\mathcal M = (\Omega, \mathscr F)$ be a measurable space and let $\mu$ and $\nu$ be $\sigma$-finite measures on $\mathcal M$...
• 20.6k
Accepted

### What is the meaning of || (double vertical bar) in this KL divergence equation?

My understanding is that the double bar emphasises that the order of the arguments matters. The reminder is perhaps helpful because KL is used much like a distance, but it's not symmetric, so it's not ...
• 4,326

• 3,104