# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

474 questions
Filter by
Sorted by
Tagged with
1 vote
23 views

### How to find the distribution that minimize the KL distance(divergence) to a known distribution?

Everyone! I'm reading the paper On test marginal versus conditional and I'm a little confused one place in page 9, when the author use "maximum likelihood projection" to get the covariance ...
19 views

### Can we train normalizing flows with Wasserstein distance?

To train flow based models, you usually either use forward or reverse kl as your loss function. My question is, can you use wasserstein distance directly as your loss function to replace kl? I have ...
• 101
27 views

### Source for KL-divergence of Beta distribution?

This post explains how to derive the Kullback-Leibler divergence between two beta distributions. https://math.stackexchange.com/questions/257821/kullback-liebler-divergence#comment564291_257821 I ...
• 612
1 vote
48 views

### Simplifying the Kullback-Leibler divergence for a sum of distributions

I want to find an approximation of a mixture of probability distributions that minimises the Kullback-Leibler divergence (KLD). I need to verify my result, as it seems suspect. We have a joint ...
• 161
34 views

### KS-Test and KL-divergence have diffrent result

It is a similar question to this but it didn't help me When KL Divergence and KS test will show inconsistent results? I have run into a situation in which I have no clue how to interpret it. I tried ...
• 73
1 vote
36 views

### Empirical error in Kullback-Leibler KL divergence estimation

In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
• 885
14 views

### When comparing two distributions, what is a 'low' value of a Kullback–Leibler divergence?

In essence how do we know a low value is good enough? Given the values could be 0 to infinity. Is a value X 'good' for any two distributions? (a little like R2 score is 'unites') Or could it be like ...
• 21
15 views

### FID as a metric to evaluate the quality of synthetic datasets (Non GAN generated) for training models for a given classification task

I am working on a problem of generating synthetic data (algorithmically by blender, not using GANs) to aid the training of some CNN for a classification ask. Ideally, I want to generate an algorithm ...
• 135
1 vote
32 views

### What does the (statistical) operator $\mathbb{D}$ usually mean? [closed]

Im reading and trying to understand the following paper on "Disentangled State Space Representations" (https://arxiv.org/pdf/1906.03255.pdf). In the derivation of the KL-loss term the ...
• 11
10 views

### What is the optimal number of bins (buckets) when calculating Population Stability Index?

I see many references to population stability index (PSI) and most all casually say to use 10 or 20 bins. Almost all examples I see for PSI use 10 bins. But is there some better method to calculate ...
27 views

### Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
• 223
21 views

69 views

• 1,679
1 vote