# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

535 questions
Filter by
Sorted by
Tagged with
20 views

### How to derive instant-dependent regret for KL-UCB bandit?

I was reading KL-UCB algorithm for bandit with Bernoulli reward from Bandit Algorithms book by Lattimore (Section 10.2), and the regret provided by the algorithm is instant-dependent and it depends on ...
• 693
129 views

### Theoretical justification for minimizing $KL(q_\phi|p)$ rather than $KL(p|q_\phi)$?

Suppose we have a true but unknown distribution $p$ over some discrete set (i.e. assume no structure or domain knowledge), and a parameterized family of distributions $q_\phi$. In general it makes ...
• 2,877
1 vote
18 views

### Difference between 3 (not PDF/CDF) distribuitions [closed]

I have multiple (>20) variables that describe 3 different objects, and I wanted to see which variables differ the most across these 3 objects. I was thinking about using KL-divergence/KS-test/...
14 views

### ShapeNet VAE KL Divergence issues

I am trying to train a VAE on shapenet but I can't seem to make it work. Any help or ideas would be highly appreciated. Now the problem is whenever I apply the KL divergence loss the network seems to ...
17 views

### Prove Decreasing Cross Entropy of outputs with Decreasing KL divergence of inputs

I am trying to prove the inequality $H(gt, y) > H(gt, y_1) > H(gt, y_2)$, given that $D_{KL}(x, x_1) > D_{KL}(x_1, x_2)$, where $y = f(x)$, $gt$ - ground truth, $D_{KL}$ - KL divergence, $H$ ...
21 views

### How do you choose to put a distribution on the right or left of KL divergence? [duplicate]

I always thought of KL divergence as a distance metric between distributions, much like Earth-Movers distance. But I can no longer ignore the asymmetry. A real distance metric is symmetric. How should ...
• 391
1 vote
46 views

### Why not use the $L^2$ norm as the difference between two probability distributions (as opposed to KL-Divergence and others) [closed]

So I was wondering why not just use: $$dist(p,q)=\bigg(\int_{x \in X} |p(x)-q(x)|^2 dx\bigg)^{1/2}$$ instead of the commonly used KL-Divergence, which isn't even a distance measure and therefore not ...
• 121
54 views

### Comparing Two Distributions of Human Estimated Probabilities

I have a two distributions of human-estimated (subjective) probabilities. Namely, let's say I have a group of meteorologists and a group of laypeople. Each person in either group estimates ...
60 views

### Negative KL Divergence estimates

I was exploring the KL Divergence and came across some research about calculating it from samples. On stack-exchange, I found out that minimising the KL Divergence is equivalent to minimising the Sum ...
• 21
1 vote
91 views

### Expectation of KL-divergence only as the log ratio of the probabilities

In the DPO paper, and in particular in the proof attached below, how can we expand the KL divergence only as the log ratio of the probabilities of the two distributions? According to the definition ...
• 111
43 views

### Comparing two two-dimensional probability distributions

I would like to compare two contour plots representing two probability distributions. These are not samples drawn from any known distribution, but the result of a physical calculation from a process ...
29 views

### Kullback-Leibler Divergence to compare two sample distributions

I have an algorithm to which I supply a specific input. The algorithm is deterministic, and will always yield the same result. I also have a random algorithm that uses random choices to speed up ...
• 101
34 views

### KL divergence between normal and skewnormal distribution

I am trying to find an analytical expression for the KL divergence between a normal distribution and a skewnormal distribution. In this paper https://www.mdpi.com/1099-4300/14/9/1606 they derive the ...
1 vote
29 views

• 2,246