# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

523 questions
Filter by
Sorted by
Tagged with
21 views

28 views

### Jeffreys divergence for a normal bivariate

I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
73 views

### "Laplace correction" in R

I am trying to understand what is the "Laplace correction" in R, in particular, when applied to this function: ...
34 views

### KL divergence for disjoint distributions

According to the article here, we have two disjoint distributions as shown below. . KL divergence for the distributions are I don't understand why denominators are 0 for both.
49 views

### Confusion with the "lower bound"-term in diffusion models

I am trying to understand the maths of diffusion models following this video explanation on youtube and this blog post. Here is what how I understood it so far: The overall goal is, that we want to ...
1 vote
47 views

### Can the average log probability score of a model be used as an approximation of the KL divergence?

I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection. I think I've grasped the concept of what would be the KL Divergence, and ...
62 views

### Should the KL loss term for a VAE be the KL-Loss of a batch's mean mu and log sigma, or is it the mean of the kl loss for each individual input image?

I've been trying to learn about Variational Autoencoders and been looking at the Keras sample implementation (https://github.com/keras-team/keras-io/blob/master/examples/generative/vae.py) I'm ...
48 views

### Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
244 views

### Conceptual questions about the proxy distribution in variational inference

I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC. My question ...
32 views

### Too large KL-divergence in training

I tried to train bayesian network using ELBO loss function. \begin{align*} \mathcal{F}(D, \theta) = KL[q(w|\theta)||P(w)] - \mathbb{E}_{q(w|\theta)}[\log p(D|w)] \end{align*} My question is, if model ...
38 views

### Kullback-Leibler divergence between product of independent gaussians and a multivariate normal distribution

what's the correct way to quantify the loss of information we have when we approximate the likelihood from multivariate normal distribution with a full covariance matrix to a product of univariate ...
1 vote
26 views

1 vote
199 views

### Understanding Objective in OpenAI InstructGPT paper?

The following objective is taken from the paper 'Training language models to follow instructions with human feedback':which is used to fine-tune the pre-trained language model using Proximal Policy ...
387 views

### Kullback–Leibler divergence between two multivariate t distributions with different degrees of freedom?

I want to calculate the Kullback–Leibler divergence between two multivariate $t$ distributions with different degrees of freedom (say $\nu_1$ and $\nu_2$), but same location and scale matrix, for ...
297 views

### What are the advantages (if any) of the Kolmogorov-Smirnov test over other tests?

Given a reference distribution and an unknown sample, we need some statistical test to determine if the unknown sample came from the reference (one-sample test), or given two samples to determine ...
1k views

### How do you find the KL Divergence between two multi-variable datasets?

Background I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
1 vote
81 views

### Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2)$$ or some variation ...
34 views

### Bayes estimate of mixture of exponential under the the Kullback-Leibler divergence loss function

Model framework: Suppose that the loss function is given by the Kullback-Leibler divergence (KLD) as follows: \begin{equation} \text{KL}(\Theta \parallel \hat{\Theta}) = \text{KL}\big(f(x;\Theta) \...
1 vote
I would like some advice or path to follow to solve the following problem. Consider a random variable $Y$ that follows a Dirichlet distribution $Y \sim Dir(\alpha)$. Let $X$ be a member of the ...