Skip to main content

Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

How to derive instant-dependent regret for KL-UCB bandit?

I was reading KL-UCB algorithm for bandit with Bernoulli reward from Bandit Algorithms book by Lattimore (Section 10.2), and the regret provided by the algorithm is instant-dependent and it depends on ...
Amin's user avatar
  • 693
5 votes
2 answers
129 views

Theoretical justification for minimizing $KL(q_\phi|p)$ rather than $KL(p|q_\phi)$?

Suppose we have a true but unknown distribution $p$ over some discrete set (i.e. assume no structure or domain knowledge), and a parameterized family of distributions $q_\phi$. In general it makes ...
user56834's user avatar
  • 2,877
1 vote
0 answers
18 views

Difference between 3 (not PDF/CDF) distribuitions [closed]

I have multiple (>20) variables that describe 3 different objects, and I wanted to see which variables differ the most across these 3 objects. I was thinking about using KL-divergence/KS-test/...
thfdealer's user avatar
0 votes
0 answers
14 views

ShapeNet VAE KL Divergence issues

I am trying to train a VAE on shapenet but I can't seem to make it work. Any help or ideas would be highly appreciated. Now the problem is whenever I apply the KL divergence loss the network seems to ...
Youssef's user avatar
0 votes
0 answers
17 views

Prove Decreasing Cross Entropy of outputs with Decreasing KL divergence of inputs

I am trying to prove the inequality $H(gt, y) > H(gt, y_1) > H(gt, y_2)$, given that $D_{KL}(x, x_1) > D_{KL}(x_1, x_2)$, where $y = f(x)$, $gt$ - ground truth, $D_{KL}$ - KL divergence, $H$ ...
user412867's user avatar
0 votes
0 answers
21 views

How do you choose to put a distribution on the right or left of KL divergence? [duplicate]

I always thought of KL divergence as a distance metric between distributions, much like Earth-Movers distance. But I can no longer ignore the asymmetry. A real distance metric is symmetric. How should ...
profPlum's user avatar
  • 391
1 vote
0 answers
46 views

Why not use the $L^2$ norm as the difference between two probability distributions (as opposed to KL-Divergence and others) [closed]

So I was wondering why not just use: $$dist(p,q)=\bigg(\int_{x \in X} |p(x)-q(x)|^2 dx\bigg)^{1/2}$$ instead of the commonly used KL-Divergence, which isn't even a distance measure and therefore not ...
Anon's user avatar
  • 121
0 votes
1 answer
54 views

Comparing Two Distributions of Human Estimated Probabilities

I have a two distributions of human-estimated (subjective) probabilities. Namely, let's say I have a group of meteorologists and a group of laypeople. Each person in either group estimates ...
phylosopher's user avatar
2 votes
0 answers
60 views

Negative KL Divergence estimates

I was exploring the KL Divergence and came across some research about calculating it from samples. On stack-exchange, I found out that minimising the KL Divergence is equivalent to minimising the Sum ...
Beetel's user avatar
  • 21
1 vote
1 answer
91 views

Expectation of KL-divergence only as the log ratio of the probabilities

In the DPO paper, and in particular in the proof attached below, how can we expand the KL divergence only as the log ratio of the probabilities of the two distributions? According to the definition ...
goodolddays's user avatar
0 votes
0 answers
43 views

Comparing two two-dimensional probability distributions

I would like to compare two contour plots representing two probability distributions. These are not samples drawn from any known distribution, but the result of a physical calculation from a process ...
lethobentho's user avatar
0 votes
0 answers
29 views

Kullback-Leibler Divergence to compare two sample distributions

I have an algorithm to which I supply a specific input. The algorithm is deterministic, and will always yield the same result. I also have a random algorithm that uses random choices to speed up ...
Robert's user avatar
  • 101
0 votes
0 answers
34 views

KL divergence between normal and skewnormal distribution

I am trying to find an analytical expression for the KL divergence between a normal distribution and a skewnormal distribution. In this paper https://www.mdpi.com/1099-4300/14/9/1606 they derive the ...
maxlman's user avatar
1 vote
0 answers
29 views

Metric for inter-model agreement?

I have a classification task, with 3 classes. A model $M$ is a classifier that outputs the probability of the input being each of these classes. For example, for input $x$, I have $M(x)=[0.1, 0.2, 0.7]...
whoisit's user avatar
  • 101
2 votes
1 answer
82 views

Z-test with no relevant sample size as I have a Gaussian probability distribution

I have two Gaussian curves, there are not samples, these are just probability distributions essentially. So I can do a Gaussian fit on them, or also a weighted average and weighted variance on the ...
Dominik Duleba's user avatar
3 votes
1 answer
85 views

Discretization of Kullback-Leibler Divergence

I have a question regarding the Kullback-Leibler divergence. I am working on a project, in which we compare numerically distributions: the empirical distribution of the data compared to an approximate ...
Louki's user avatar
  • 420
0 votes
0 answers
62 views

Confused on Kullback-Leibler divergence being invoked without proper definition

I am trying to understand how authors of the DDPM paper in appendix A, made the leap from equation 21 to equation 22. Specifically, it is not clear to me how they managed to convert the first term of ...
Spacey's user avatar
  • 1,805
0 votes
0 answers
151 views

Is it better to use KL Divergence for soft labels instead of cross entropy?

So I was going through this paper: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2017) by Junnan Li et al., in this they perform contrastive loss using KL ...
mutli-arm-bandit's user avatar
0 votes
0 answers
16 views

Computational issue for finding the JS divergence

I had two datasets with 144 points in 2 dimension then I used he sklearn library to fit the GMM and that fits well , I checked for the BIC values while fitting the model and chooses the model with low ...
tiger's user avatar
  • 1
2 votes
0 answers
70 views

Proving Monotonic Decrease of Kullback-Leibler Divergence in Iterative Method for Stationary Distribution Estimation

Introduction Consider a well-behaved Markov chain with desirable properties (irreducible, aperiodic, positive recurrent), characterized by a transition matrix $P$ and a stationary distribution $\pi$. ...
Chia's user avatar
  • 43
4 votes
0 answers
27 views

Why do model selection criteria (xICs, etc) not explicitly incorporate a loss function?

Model Selection and Multimodel Inference by Burnham and Anderson notes that TIC, AIC, AICc and QAICc are based on K-L distance between a given model and true model. Also BIC is in a sense based on ...
Mohan's user avatar
  • 939
0 votes
0 answers
79 views

Conditions for existence of KL divergence and unique minimum

Consider two probability density function g(y) and f(y: $\theta$), $\theta \in \Theta$. The KL divergence of f and g is defined by $$ D_{KL}(g|f) := \int \log \frac{g(y)}{f(y: \theta)} \, dy = \...
asdfasdf kansdf's user avatar
0 votes
0 answers
27 views

How to measure the difference between two distributions of the same family?

Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
Value_Investor's user avatar
8 votes
2 answers
123 views

KL Divergence, which pdf is which?

Let $P(x)$ and $Q(x)$ be two pdfs. Let us say that $P(x)$ is the original baseline distribution and $Q(x)$ is the model (or estimate) distribution. I wanted to take the KL Divergence (or the 'distance'...
cgo's user avatar
  • 9,217
2 votes
1 answer
48 views

Pinsker-type inequality for moments?

Let $f_1$, $f_2$ be two discrete probability distributions. By Pinsker's inequality, the Kullback-Leibler divergence $D(f_1||f_2)$ sets an upper bound on the total variation distance between the two ...
Luis Mendo's user avatar
  • 1,099
0 votes
0 answers
20 views

KL divergence as minimum patch size for data differencing?

The Wikipedia article on KL divergence mentions a link with data differencing. Directly quoting Wikipedia (as of 2023/11/01): Just as absolute entropy serves as theoretical background for data ...
pppqqq's user avatar
  • 151
0 votes
0 answers
101 views

Jensen-Shannon Distance - Estimating the discrete probability distribution

In my current research, I am working with a comprehensive dataset that captures human mobility patterns across different locations. My specific focus is on quantifying the similarity between these ...
Negin Zarbakhsh's user avatar
0 votes
0 answers
70 views

KL Divergence in VAE [duplicate]

The basic KL-Divergence between two distributions is as: $KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
Martin Perry's user avatar
2 votes
1 answer
92 views

Lower bound KL distance of non linear transform of a Gaussian with a family of mean zero Gaussian

Let $X \sim \mathcal{N}(0, \sigma_x^2)$ and let $f :\mathbb{R} \to \mathbb{R}$ be a smooth nonlinear transformation such that $\mathbb{E}[f(X)]=0$. I am wondering what kind of restrictions one can put ...
Abm's user avatar
  • 372
3 votes
1 answer
46 views

Why $\nabla_{\theta} \log p(y;\theta) = \frac{\nabla_{\theta}p(y;\theta)}{p(y;\theta)}$?

I'm solving a problem for cs 229, problem description in below when i check the answer it mentioned the given equation, but I don't undertand why is that. I want to know why, anyone give me some sort ...
Yiffany's user avatar
  • 185
0 votes
0 answers
58 views

What is the distribution of the sample (empirical) KL divergence and can we use bootstrapping to approximate it?

Let $p, q$ be two distributions, and suppose one can evaluate them pointwise. Let $\{x_i \}_{i = 1}^n$ be an IID sample from $p$, and let $KL_n = \frac{1}{n}\sum_i \log\left[ \frac{p(x_i)}{q(x_i)} \...
travelingbones's user avatar
0 votes
1 answer
50 views

Jeffreys divergence for a normal bivariate

I am reading the book Information Theory and Statistics of S. Kullback. In page 8 ((4.3) it is shown that the KL divergence between the joint bivariate normal and the product of the corresponding ...
noosdev's user avatar
  • 11
3 votes
1 answer
155 views

"Laplace correction" in R

I am trying to understand what is the "Laplace correction" in R, in particular, when applied to this function: ...
Ommo's user avatar
  • 270
1 vote
1 answer
92 views

Can the average log probability score of a model be used as an approximation of the KL divergence?

I'm reading the Chapter 7 of Statistical Rethinking (2nd), where the author delves into information theory and model selection. I think I've grasped the concept of what would be the KL Divergence, and ...
Idervas's user avatar
  • 107
0 votes
0 answers
74 views

Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
YP BARRY's user avatar
2 votes
1 answer
269 views

Conceptual questions about the proxy distribution in variational inference

I am trying to implement a variational extension of some kind of Bayesian network estimation method. The main goal is to improve speed, since the current method is pretty slow due to MCMC. My question ...
Mangnier Loïc's user avatar
1 vote
0 answers
137 views

Too large KL-divergence in training

I tried to train bayesian network using ELBO loss function. \begin{align*} \mathcal{F}(D, \theta) = KL[q(w|\theta)||P(w)] - \mathbb{E}_{q(w|\theta)}[\log p(D|w)] \end{align*} My question is, if model ...
Kim Seong Hyeon's user avatar
0 votes
0 answers
127 views

Kullback-Leibler divergence between product of independent gaussians and a multivariate normal distribution

what's the correct way to quantify the loss of information we have when we approximate the likelihood from multivariate normal distribution with a full covariance matrix to a product of univariate ...
Alucard's user avatar
  • 325
1 vote
0 answers
29 views

Derivation of the target upper-bound with Jensen's inequality in 'Divide-and-Conquer RL'

I am currently stuck in one line in the paper named "Divide-and-Conquer Reinforcement Learning"(Ghosh et. al., 2018 ICLR). It is the equation (1) in the page 4 which is like below. $$E_\pi[...
KCLEE's user avatar
  • 11
24 votes
2 answers
2k views

Why are there extra terms $-p_i+q_i$ in SciPy's implementation of Kullback-Leibler divergence?

Why do some definitions of the Kullback-Leibler divergence include extra terms $-p_i + q_i$? For example, kl_div() (in the Python ...
Igor F.'s user avatar
  • 9,408
3 votes
2 answers
355 views

Replacing the KL-divergence term in a VAE with parameter regularization

When training a VAE, one aim to optimize function $\mathcal{L}$, defined as: $$\mathcal{L}\left(\theta,\phi; \mathbf{x}^{(i)}\right) = - D_{KL}\left(q_\phi(\mathbf{z}|\mathbf{x}^{(i)}) || p_\theta(\...
Asterion's user avatar
  • 936
3 votes
1 answer
300 views

Understanding Objective in OpenAI InstructGPT paper?

The following objective is taken from the paper 'Training language models to follow instructions with human feedback':which is used to fine-tune the pre-trained language model using Proximal Policy ...
Tinatim's user avatar
  • 131
9 votes
1 answer
650 views

Kullback–Leibler divergence between two multivariate t distributions with different degrees of freedom?

I want to calculate the Kullback–Leibler divergence between two multivariate $t$ distributions with different degrees of freedom (say $\nu_1$ and $\nu_2$), but same location and scale matrix, for ...
Student's user avatar
  • 93
14 votes
2 answers
738 views

What are the advantages (if any) of the Kolmogorov-Smirnov test over other tests?

Given a reference distribution and an unknown sample, we need some statistical test to determine if the unknown sample came from the reference (one-sample test), or given two samples to determine ...
Number's user avatar
  • 1,040
5 votes
2 answers
3k views

How do you find the KL Divergence between two multi-variable datasets?

Background I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
Connor's user avatar
  • 635
1 vote
1 answer
121 views

Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$ or some variation ...
pyrrosk's user avatar
  • 33
0 votes
0 answers
38 views

Bayes estimate of mixture of exponential under the the Kullback-Leibler divergence loss function

Model framework: Suppose that the loss function is given by the Kullback-Leibler divergence (KLD) as follows: \begin{equation} \text{KL}(\Theta \parallel \hat{\Theta}) = \text{KL}\big(f(x;\Theta) \...
Statistics 's user avatar
1 vote
0 answers
96 views

Advice on how to solve a constrained KL Divergence problem between a Dirichlet and a Logistic Normal

I would like some advice or path to follow to solve the following problem. Consider a random variable $Y$ that follows a Dirichlet distribution $Y \sim Dir(\alpha)$. Let $X$ be a member of the ...
Javier's user avatar
  • 76
2 votes
1 answer
92 views

How to write a Kullback–Leibler divergence in mathematically correct notation?

Let we have random variables $X$, $Y$, $Z$ and a function $q: Z \to \mathbb{R}$. How to write a Kullback–Leibler divergence between $q(Z)$ and $p(Z|X)$ mathematically correct? If we write $D_{\text{...
Oleg S's user avatar
  • 29
2 votes
0 answers
35 views

Symmetrization of the G-test statistic?

G-test statistic is defined as $$ G(O,E) = 2 \sum_{i=1}^N O_i \ln \frac{O_i}{E_i} $$ and $G \sim \chi^2_{N-1}$ asymptotically. How is $$ \begin{eqnarray} S &=& G(O,E)+G(E,O) \\ &=& ...
sds's user avatar
  • 2,246

1
2 3 4 5
11