Questions tagged [kullback-leibler]
An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.
523
questions
0
votes
0
answers
21
views
KL Divergence in VAE [duplicate]
The basic KL-Divergence between two distributions is as:
$KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
- 225
2
votes
1
answer
70
views
Lower bound KL distance of non linear transform of a Gaussian with a family of mean zero Gaussian
Let $X \sim \mathcal{N}(0, \sigma_x^2)$ and let $f :\mathbb{R} \to \mathbb{R}$ be a smooth nonlinear transformation such that $\mathbb{E}[f(X)]=0$. I am wondering what kind of restrictions one can put ...
- 352
2
votes
1
answer
36
views
Why $\nabla_{\theta} \log p(y;\theta) = \frac{\nabla_{\theta}p(y;\theta)}{p(y;\theta)}$?
I'm solving a problem for cs 229, problem description in below when i check the answer it mentioned the given equation, but I don't undertand why is that.
I want to know why, anyone give me some sort ...
- 135
0
votes
0
answers
22
views
What is the distribution of the sample (empirical) KL divergence and can we use bootstrapping to approximate it?
Let $p, q$ be two distributions, and suppose one can evaluate them pointwise.
Let $\{x_i \}_{i = 1}^n$ be an IID sample from $p$, and let $KL_n = \frac{1}{n}\sum_i \log\left[ \frac{p(x_i)}{q(x_i)} \...
- 351
0
votes
1
answer
28
views
Jeffreys divergence for a normal bivariate
I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
- 11
2
votes
1
answer
73
views
"Laplace correction" in R
I am trying to understand what is the "Laplace correction" in R, in particular, when applied to this function:
...
- 168
0
votes
0
answers
34
views
KL divergence for disjoint distributions
According to the article here, we have two disjoint distributions as shown below.
.
KL divergence for the distributions are
I don't understand why denominators are 0 for both.
- 483
0
votes
0
answers
49
views
Confusion with the "lower bound"-term in diffusion models
I am trying to understand the maths of diffusion models following this video explanation on youtube and this blog post.
Here is what how I understood it so far:
The overall goal is, that we want to ...
- 1
1
vote
1
answer
47
views
Can the average log probability score of a model be used as an approximation of the KL divergence?
I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection.
I think I've grasped the concept of what would be the KL Divergence, and ...
- 97
0
votes
0
answers
62
views
Should the KL loss term for a VAE be the KL-Loss of a batch's mean mu and log sigma, or is it the mean of the kl loss for each individual input image?
I've been trying to learn about Variational Autoencoders and been looking at the Keras sample implementation (https://github.com/keras-team/keras-io/blob/master/examples/generative/vae.py)
I'm ...
- 121
0
votes
0
answers
48
views
Which metric to compare two probability density?
I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
- 1
2
votes
1
answer
244
views
Conceptual questions about the proxy distribution in variational inference
I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC.
My question ...
- 513
0
votes
0
answers
32
views
Too large KL-divergence in training
I tried to train bayesian network using ELBO loss function.
\begin{align*}
\mathcal{F}(D, \theta) = KL[q(w|\theta)||P(w)] - \mathbb{E}_{q(w|\theta)}[\log p(D|w)]
\end{align*}
My question is, if model ...
0
votes
0
answers
38
views
Kullback-Leibler divergence between product of independent gaussians and a multivariate normal distribution
what's the correct way to quantify the loss of information we have when we approximate the likelihood from multivariate normal distribution with a full covariance matrix to a product of univariate ...
- 325
1
vote
0
answers
26
views
Derivation of the target upper-bound with Jensen's inequality in 'Divide-and-Conquer RL'
I am currently stuck in one line in the paper named "Divide-and-Conquer Reinforcement Learning"(Ghosh et. al., 2018 ICLR). It is the equation (1) in the page 4 which is like below.
$$E_\pi[...
- 11
23
votes
2
answers
2k
views
Why are there extra terms $-p_i+q_i$ in SciPy's implementation of Kullback-Leibler divergence?
Why do some definitions of the Kullback-Leibler divergence include extra terms $-p_i + q_i$? For example, kl_div() (in the Python ...
- 8,448
0
votes
0
answers
56
views
How to choose the order of distributions in KL divergence
While building ML model I'm facing a covariate shift detection, so I need to compare old labelled data and new unlabelled data. I'm planning to use KL divergence to quantify the distances between ...
- 433
0
votes
0
answers
13
views
Mean and variance of the G-test of goodness of fit?
Let $\mathcal{O}$ (for observed) and $\mathcal{E}$ (for expected) be two $D$-variate multinomial distributions, such that $\sum_{d=1}^{D}\mathcal{O}_{d}=1$ and $\sum_{d=1}^{D}\mathcal{E}_{d}=1$. The G-...
- 545
3
votes
2
answers
143
views
Replacing the KL-divergence term in a VAE with parameter regularization
When training a VAE, one aim to optimize function $\mathcal{L}$, defined as:
$$\mathcal{L}\left(\theta,\phi; \mathbf{x}^{(i)}\right) = - D_{KL}\left(q_\phi(\mathbf{z}|\mathbf{x}^{(i)}) || p_\theta(\...
- 926
1
vote
1
answer
199
views
Understanding Objective in OpenAI InstructGPT paper?
The following objective is taken from the paper 'Training language models to follow instructions with human feedback':which is used to fine-tune the pre-trained language model using Proximal Policy ...
- 111
9
votes
1
answer
387
views
Kullback–Leibler divergence between two multivariate t distributions with different degrees of freedom?
I want to calculate the Kullback–Leibler divergence
between two multivariate $t$ distributions with different degrees of freedom (say $\nu_1$ and $\nu_2$), but same location and scale matrix, for ...
- 93
13
votes
2
answers
297
views
What are the advantages (if any) of the Kolmogorov-Smirnov test over other tests?
Given a reference distribution and an unknown sample, we need some statistical test to determine if the unknown sample came from the reference (one-sample test), or given two samples to determine ...
- 980
4
votes
2
answers
1k
views
How do you find the KL Divergence between two multi-variable datasets?
Background
I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
- 369
1
vote
1
answer
81
views
Calculating KL divergence with entropy and cross entropy for VAEs
When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is:
$$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$
or some variation ...
- 23
0
votes
0
answers
34
views
Bayes estimate of mixture of exponential under the the Kullback-Leibler divergence loss function
Model framework:
Suppose that the loss function is given by the Kullback-Leibler divergence (KLD) as follows:
\begin{equation}
\text{KL}(\Theta \parallel \hat{\Theta}) = \text{KL}\big(f(x;\Theta) \...
- 143
1
vote
0
answers
40
views
Advice on how to solve a constrained KL Divergence problem between a Dirichlet and a Logistic Normal
I would like some advice or path to follow to solve the following problem.
Consider a random variable $Y$ that follows a Dirichlet distribution $Y \sim Dir(\alpha)$. Let $X$ be a member of the ...
2
votes
1
answer
70
views
How to write a Kullback–Leibler divergence in mathematically correct notation?
Let we have random variables $X$, $Y$, $Z$ and a function $q: Z \to \mathbb{R}$. How to write a Kullback–Leibler divergence between $q(Z)$ and $p(Z|X)$ mathematically correct?
If we write $D_{\text{...
- 29
1
vote
0
answers
27
views
Symmetrization of the G-test statistic?
G-test statistic is defined as
$$
G(O,E) = 2 \sum_{i=1}^N O_i \ln \frac{O_i}{E_i}
$$
and $G \sim \chi^2_{N-1}$ asymptotically.
How is
$$
\begin{eqnarray}
S &=& G(O,E)+G(E,O) \\
&=& ...
- 2,196
2
votes
0
answers
121
views
Why is the Wasserstein distance not used in Variational Inference
I just started learning the concept of variational inference in the context of variational Autoencoder, so please excuse me if the answer is obvious. I would like to know why traditionally, KL-...
- 21
1
vote
0
answers
207
views
How exactly does the weight applied to the KL divergence in $\beta$-VAE lead to disentanglement?
In "β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK", the Introduction states that "With β > 1 the model is pushed to learn a more efficient latent ...
- 23
2
votes
0
answers
35
views
Minimizer of Kullback-Leibler divergence of non-symmetric distributions
I am working on a convergence result for an economics paper but am struggling with the following:
Suppose you have two distributions, $f(a)$ and $g(a)$ of the same non-symmetric family. Suppose that ...
- 21
1
vote
1
answer
75
views
Derivation of ELPD from KL Divergence
I am currently learning about how to estimate the predictive accuracy of (bayesian) models. See here: https://bebi103b.github.io/lessons/18/model_comparison.html#id4
I need help regarding the last ...
- 21
0
votes
0
answers
43
views
How to define the relation between the Fisher metric and KL divergence in the following case
Let we have a point cloud data consists of exactly $n$ distinct points in $\mathbb{R^d}$ that each each point clod is of the form $X=\{x_1,...,x_n/ x_i\in \mathbb{R^d},x_{i}\neq x_{j},i\neq j\}$. The ...
- 133
2
votes
0
answers
119
views
Understanding intuitive difference between KL divergence and Cross entropy
I know there are related questions already asked, for example this one.
I also know the following:
KL divergence $D_{KL}(P\Vert Q)$ is given as:
$$\begin{align}
D_{KL}(P\Vert Q) & = -\sum_xP(x)\...
- 215
2
votes
1
answer
258
views
Notation confusion regarding Expectation in Kullback-Leibler divergence definition
KL-divergence is often defined as
$$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$
I don't quite ...
- 1,174
0
votes
0
answers
17
views
Understanding predictive distribution expression from an information theory perspective
Suppose we are given a set of $M$ models $f^{1}(\mathbf{x}), \cdots, f^{M}(\mathbf{x})$ each taking a vector $\mathbf{x}$ as input and outputting probabilities. We also have access to the one hot ...
- 546
0
votes
0
answers
20
views
How adding sparsity term in loss function of sparse autoencoder in-actives the hidden node?
I am working on a Sparse Autoencoder but Andrew NG's notes are hard to understand. My question is about the following equation: Loss Function. image
In sparse autoencoder, the goal is to inactive some ...
1
vote
0
answers
37
views
How can I derive this equation?
I am reading Understanding Diffusion Models: A Unified Perspective, but I am confused about equation 45. Concretely, the consistency term:
$$
\sum_{t=1}^{T-1} \mathbb{E}_{q(x_{t-1},x_t,x_{t+1}|x_0)}\...
- 121
0
votes
0
answers
300
views
What is the closed-form of the KL-Divergence between two relaxed Bernoulli distributions?
I've seen in multiple papers that use a relaxation of the Bernoulli distribution as defined in Maddison et. al (here it is referred to as Binary Concrete) and they say that a closed form solution for ...
1
vote
0
answers
27
views
Expectation of conditional KLs
I was reading how not to train your generative model paper. I don’t quite understand how the simplification from equation 4 to equation 5 can be right and based on my calculations it should be wrong. ...
0
votes
0
answers
42
views
Difference between two normal distributions [duplicate]
For the discrete case (in programming) the answer is clear to me.
I would put my samples into the Kulback-Leibler-Divergence or maybe cross entropy.
In my case I have pairs of normal distributions or ...
- 103
0
votes
0
answers
62
views
Expected KL divergence
The KL divergence between two univariate gaussian distributions $P_{\mu_P, \sigma^2_P}$ and $Q_{\mu_Q, \sigma^2_Q}$ is given by,
\begin{align}
\textrm{D}_{\textrm{KL}}(P||Q) &= \frac{1}{2} \...
- 11
0
votes
1
answer
107
views
KL divergence between GPs undefined?
In Theorem 1 in Sun et al, Functional Variational Bayesian Neural Networks, 2019, the authors state the the KL divergence between stochastic processes in the supremum over KL divergence between ...
- 33
0
votes
0
answers
72
views
KL divergence between gaussian with uniform prior
I have 2 normal distributions $\mathcal{N}(\mu_1, \mathbb{I}_d)$ where $\mu_1$ is a fixed vector in $\mathbb{R}^d$ and $\mathcal{N}(\mu_2, \mathbb{I}_d)$, where $\mu_2$ is $\mu_1 + V$, where $V$ is ...
- 183
0
votes
0
answers
31
views
How to compare two very large vectors that each represent a probability mass function?
As far as I know, given two vectors that each represent a probability mass function, their difference can be measured using Euclidean distance, Kullback–Leibler divergence, cross entropy and so on. ...
- 131
5
votes
1
answer
1k
views
Kullback–Leibler divergence between two normal distributions
I am currently reading 'Dive into Deep Learning' and right now I am trying to improve my intuition for the Kullback–Leibler divergence. I get the basic idea, why this metric is not symmetric, however, ...
- 505
0
votes
0
answers
99
views
Familiarity with KL Divergence Vectors
a = [2,8,10,0,3,0,0]
b = [0,0,3,4,2,0,0]
I would like to find out how much different the b distribution is from the ...
- 101
0
votes
0
answers
13
views
How is KL divergence (p||q) different from KL divergence (q||p)? [duplicate]
I understand the difference in formula, but I am not being able to understand the difference intuitively.
Thank you!
- 411
0
votes
0
answers
68
views
KL divergence between 2 classes samples of a dataset
I want to identify how well the samples in the dataset are differentiated. For that, my assumption is the KL-divergence between the samples of different classes will be larger. Am I correct, can I use ...
- 319
0
votes
0
answers
11
views
Does the following "approach" for integrating over the data space makes sense?
Suppose that we have the following posterior and prior distributions
$p(\mu|x,m_{1}(x),s_{1}(x)) = Normal(\mu;m_{1}(x),s_{1}(x))$ and $p(\mu|m_{2},s_{2})$
The $m_{1}(x),s_{1}(x)$ indicate that the ...
- 2,142