# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

387 questions
Filter by
Sorted by
Tagged with
41 views

### How to compare if two multinomial distributions are significantly different

We can use T test to check if two proportions are significantly different. Similarly is there a way to test if two multinomial distributions or "2 samples with more than 2 unique values" are ...
25 views

### KL-Divergence to evaluate regression model performance [duplicate]

Would it make sense to use KL-Divergence to measure the difference in predictions versus ground truth for a regression problem? I've tuned four models and serve the average as a prediction in the ...
28 views

### KL Divergence Between Ground Truth and Prediction

I've got four (non-linear, tree-based) models in production and using the average of them as the served prediction. We get ground truth data immediately. During training the optimized candidate models ...
8 views

### Rising “sub-losses” when optimizing sum of losses with ADAM optimizer (KL divergence, neg. loglik)

I am optimizing the ELBO as part of the variational inference using a neural network for a dynamic topic model (D-ETM). The loss being optimized is the sum of the negative log-likelihood and KL ...
17 views

### If entropy is the underlying measure for KL-divergence, what is the underlying measure for the Wasserstein distance?

If entropy is the basis measure underlying KL-divergence (aka relative entropy), what is the basis measure underlying the Wasserstein distance?
47 views

165 views

### Are Mutual Information and Kullback–Leibler divergence equivalent?

From my readings, I understand that: Mutual information $\mathit{(MI)}$ is a metric as it meets the triangle inequality, non-negativity, indiscernability and symmetry criteria. The Kullback–Leibler ...
21 views

### Does it make sense to use the KL-divergence between joint distributions of synthetic and real data, as a evaluation metric?

The KL-divergence is defined as: $D_{KL}(p(x_1)∥q(x_1))=\sum p(x_1)\, \log \Big( \dfrac{p(x_1)}{q(x_1)} \Big)$ I consider the Kullback-Leibler (KL) divergence as a performance metric for data ...
68 views

### Should reconstruction loss be computed as sum or average over input for variational autoencoders?

I am following this variational autoencoder tutorial: https://keras.io/examples/generative/vae/. I have included the loss computation part of the code below. I know VAE's loss function consists of the ...
29 views

### Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?

We know KL-divergence is sometimes expressed like this: which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
18 views

### Measurement to compare probability distributions

I have a set of probability distribution that I want to compare. Right now, I'm relying on (subjective) ocular evaluations of the plot. Is there any more appropriate statistical measurement? ...
60 views

### Kullback-Leibler divergence and marginals

Let $P(x,y)$ and $Q(x,y)$ be two probability distributions, with marginals $$P(x)=\sum_y P(x,y),\quad P(y)=\sum_x P(x,y),\quad Q(x)=\sum_y Q(x,y),\quad Q(y)=\sum_x Q(x,y)$$ What is the relation ...
18 views

85 views

### KL-divergence: P||Q vs. Q||P

Assume, that we have several data generating measures $P_{1}, \dots, P_{k}$ and $Q$, all defined on the same probability space. Next, assume, we have the same amount of independently sampled data from ...
15 views

52 views

### Laplace and Normal Distribution Cross Entropy

I need the following integral and struggle with calculating it or finding a citable source. $$\int_{-\infty}^{\infty}(x-\mu)^2\exp\!\left(-\frac{|x-\nu|}{\tau}\right)dx.$$ Background: I want to find ...
90 views

### KL Divergence Normal and Laplace densities

I want to calculate the KL-Divergence between a Laplacian density g and a normal density f. I can decompose $KL(G|F)$ to $\mathbb{E}_g[\log g(X)]-\mathbb{E}_g[\log f(X)]$. I am already stuck with my ...
44 views

### Transformation of discrete KL-Divergence in Continuous KL-Divergence possible?

I Use Python and the following definition of KL-Divergence def kl_divergence(p, q): return np.sum(np.where(p != 0, p * np.log(p / q), 0)) to calculate the ...
28 views

### what is -0.5 in VAE loss function with KL term

The VAE loss is composed of two terms: Reconstruction loss KLD loss in the implementation there is -0.5 applied to KLD loss. Kindly let me know what is this -0.5
16 views

### How to minimize KL Divergence in VAE loss?

I am training VAE autoencoder model. VAE has loss combining MSE+KL divergence. When I train the model, KL loss is increasing over or near 100 while MSE loss is decreasing. So, can anyone tell me what ...
32 views

### Should I use the sum of KL divergences for multi-objective model selection?

I have a model implemented in Python with 2 free parameters. I would like to find the parameter values that provide the best fit to empirical data comprising of response times and accuracy of human ...
### Can $G^2$ statistic in log-linear model for contingency tables be negative?
Can $G^2$ statistic of log-linear (unsaturated) model in contingency tables be negative? Since saturated model with perfect fit has $G^2=0$ I don't think the unsaturated models can get negative $G^2$. ...