Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
1 vote
0 answers
23 views

How to find the distribution that minimize the KL distance(divergence) to a known distribution?

Everyone! I'm reading the paper On test marginal versus conditional and I'm a little confused one place in page 9, when the author use "maximum likelihood projection" to get the covariance ...
user avatar
0 votes
0 answers
19 views

Can we train normalizing flows with Wasserstein distance?

To train flow based models, you usually either use forward or reverse kl as your loss function. My question is, can you use wasserstein distance directly as your loss function to replace kl? I have ...
user avatar
  • 101
0 votes
0 answers
27 views

Source for KL-divergence of Beta distribution?

This post explains how to derive the Kullback-Leibler divergence between two beta distributions. https://math.stackexchange.com/questions/257821/kullback-liebler-divergence#comment564291_257821 I ...
user avatar
1 vote
0 answers
48 views

Simplifying the Kullback-Leibler divergence for a sum of distributions

I want to find an approximation of a mixture of probability distributions that minimises the Kullback-Leibler divergence (KLD). I need to verify my result, as it seems suspect. We have a joint ...
user avatar
  • 161
0 votes
0 answers
34 views

KS-Test and KL-divergence have diffrent result

It is a similar question to this but it didn't help me When KL Divergence and KS test will show inconsistent results? I have run into a situation in which I have no clue how to interpret it. I tried ...
user avatar
  • 73
1 vote
1 answer
36 views

Empirical error in Kullback-Leibler KL divergence estimation

In computing the Kullback-Leibler KL divergence $D(P\|Q)$ from an empirical data, it may happen that $Q(x)=0<P(x)$ at some sample point $x$ due to data error and $D(P\|Q)=\infty$. What are some ...
user avatar
  • 885
0 votes
0 answers
14 views

When comparing two distributions, what is a 'low' value of a Kullback–Leibler divergence?

In essence how do we know a low value is good enough? Given the values could be 0 to infinity. Is a value X 'good' for any two distributions? (a little like R2 score is 'unites') Or could it be like ...
user avatar
  • 21
0 votes
0 answers
15 views

FID as a metric to evaluate the quality of synthetic datasets (Non GAN generated) for training models for a given classification task

I am working on a problem of generating synthetic data (algorithmically by blender, not using GANs) to aid the training of some CNN for a classification ask. Ideally, I want to generate an algorithm ...
user avatar
  • 135
1 vote
0 answers
32 views

What does the (statistical) operator $\mathbb{D}$ usually mean? [closed]

Im reading and trying to understand the following paper on "Disentangled State Space Representations" (https://arxiv.org/pdf/1906.03255.pdf). In the derivation of the KL-loss term the ...
user avatar
  • 11
0 votes
0 answers
10 views

What is the optimal number of bins (buckets) when calculating Population Stability Index?

I see many references to population stability index (PSI) and most all casually say to use 10 or 20 bins. Almost all examples I see for PSI use 10 bins. But is there some better method to calculate ...
user avatar
0 votes
0 answers
27 views

Why is KL divergence used as a measure of closeness in variational inference?

I am curious why KL divergence is the standard measure of (dis)similarity used in VI while it is not even a proper metric (asymmetric and does not satisfy triangle inequality).
user avatar
  • 223
0 votes
0 answers
21 views

The lower bound of K-L divergence of a mixture

I'm wondering if there is a lower bound for a mixture when each single component K-L divergence in the mixture is lower bounded by some constants. Let $$D(p||q)=\int p(x)\log \frac{p(x)}{q(x)}dx$$ If $...
user avatar
0 votes
1 answer
57 views

minimizing kullback-Leiber divergence

This paper is showing how KL divergence can be minimized by matching the expected values of the sufficient statistics. More precisely, For any distribution p of the exponential family with pdf: $$ p_{\...
user avatar
  • 117
0 votes
0 answers
20 views

Bayes Risk with Kullback Leibler divergence

Consider two random variables $X\sim N(\theta, 1)$ and $Y\sim N(\theta, r)$ with $\theta\sim \pi$. Then the Kullback Leibler divergence between the original joint density $f_{X, Y}(x,y)=\phi(x-\theta)\...
user avatar
  • 211
1 vote
0 answers
17 views

Why KL-Divergence is coming out to be reversed? [closed]

I am trying to assess the difference between my prior distribution and the posterior distribution by calculating the KL divergence metric. I am using the following KL divergence formula - KL = np.sum(...
user avatar
1 vote
0 answers
49 views

Taking the KL divergence between a normal distribution and an empirical distribution

If we have a multivariate Gaussian $q(\mathbf{a}) \sim \mathcal{N}(\mathbf{\mu}, \Sigma) $ and an empirical distribution $P(\mathbf{a} ) = \sum_{k=1}^{K} p_k \ \delta(\mathbf{a} - \mathbf{a_k})$ and I ...
user avatar
  • 11
0 votes
0 answers
8 views

Significance of difference between empirical probability distributions

I have 1,000 pairs of probability distributions. Each pair $i$ consists of two discrete probability distributions that are measured empirically using $n_i$ and $m_i$ samples. i.e. every probability ...
user avatar
0 votes
0 answers
15 views

Analogous information matrix and divergence for the Bhattacharyya bound

In the case of Cramér-Rao lower bound (CRLB), the Fisher information matrix (FIM) is obtained from the K-L divergence (KLD), i.e. $D(p_\theta\|p_\theta') = \int p_\theta(x)\log\frac{p_\theta(x)}{p_{\...
user avatar
  • 153
2 votes
2 answers
102 views

Why is mutual information symmetric?

I know that mutual information is the Kullback-Leibler divergence between $p(x,y)$ and $p(x)p(y)$. But mutual information is also described as the amount of entropy lost (or, in another sense, the ...
user avatar
  • 147
0 votes
1 answer
53 views

Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
user avatar
  • 3
1 vote
1 answer
89 views

Why does mutual information use KL divergence?

Mutual information between a pair of random variables $X,Y$ having joint distribution $P_{(X,Y)}$ and marginal distributions $P_X,P_Y$ respectively is defined as $$I(X,Y)\equiv D_{\text{KL}}(P_{(X,Y)}\...
user avatar
0 votes
0 answers
11 views

Kullback Leibler divergence between normal and Cauchy distributions? [duplicate]

Let $\phi(x)$ be the standard normal probability density function and $f(x)$ be the Cauchy probability density function. How can I calculate the Kullback-Leibler divergences between $\phi$ and $f$. ...
user avatar
  • 1
2 votes
1 answer
69 views

Computing KL divergence between uniform and multivariate Gaussian

Another post has addressed the fact that KL divergence is defined between a uniform distribution and a Gaussian distribution $$D_{\text{KL}}(\mathcal{U}(x) \parallel \mathcal{N}(x \mid \mu, \Sigma)) = ...
user avatar
0 votes
0 answers
33 views

Quartic From Gaussian Expectation

$x$ is $d$-dimensional vector, Gaussian distributed with mean $\mu$ and covariance $\Sigma$. I want to simplify the same expression attached below but replacing $x$ with $x^2$ in the beginning of the ...
user avatar
0 votes
2 answers
147 views

Does it make sense to have KL divergence between 2 distributions associated with 2 different random variables? [duplicate]

Does it make sense to have KL divergence between 2 PMFs/PDFs/etc associated with two different random variables? Here is a definition: "Definition The relative entropy or Kullback-Leibler ...
user avatar
  • 93
0 votes
0 answers
11 views

Random variables in KL divergence [duplicate]

Here is a definition from wikipedia: "For discrete probability distributions $P$ and $Q$ defined on the same probability space, $\mathcal{X}$, the relative entropy from $Q$ to $P$ is defined ${ }^...
user avatar
  • 93
1 vote
1 answer
49 views

For KL divergence, would we consider two distribution has same random variable?

from the formula of KL divergence $$ KL(P||Q) = \int_{-\infty}^{\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right)\operatorname{d}x$$ I feel the $P$ and $Q$ has the same random variable, but, the ...
user avatar
1 vote
0 answers
21 views

What determines performance in recoverying K in Gaussian Mixture Model?

My question is about what determines how hard it is to recover the number of components $K$ in a Gaussian mixture model (GMM), e.g. with the EM-algorithm. For simplicity, let's consider the case in ...
user avatar
  • 653
0 votes
0 answers
25 views

Negative KL divergence for train_test_split in sklearn for y_train and y_val

So, I am trying to understand if I have fair split of my train and val sets using train_test_split of sklearn, so I decided to run the KL divergence and JS div tests and I get the following results. ...
user avatar
2 votes
1 answer
55 views

Intuition of using p(x) (true distribution probability) in KL Divergence definition

We all know that $D(p||q) = \sum_x p(x)log\frac{p(x)}{q(x)}$ and it is used to quantify the difference between the true distribution p and the observed distribution q. However, I do not get the ...
user avatar
  • 21
3 votes
1 answer
94 views

What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age). I would ...
user avatar
0 votes
0 answers
40 views

Threshold of KL divergence for detecting data drift

While I've come across posts introducing KL divergence as a mean for data drift detections. However, I fail to observe any on of them suggesting beyond what threshold value of KL divergence should we ...
user avatar
  • 135
0 votes
0 answers
69 views

Maximizing the mutual information related to minimizing KL divergence?

Let's say we are drawing two random variables from two distributions, like x~p and z~q, then maximizing mutual information I(x;z) leads to decreasing KL divergence D_kl(p|q) ? This sounds correct to ...
user avatar
  • 11
0 votes
0 answers
28 views

Compute KL divergence, given mean and variance arrays

Given that my knowledge of statistics is fairly minimal. Say i performed a MCS (Monte Carlo Simulation) with a large sample size on a function and this is treated as a reference solution. Then I have ...
user avatar
  • 43
0 votes
0 answers
29 views

Information Gain as a measure of term importance

I was watching the talk Beyond TF-IDF: Why, What and How. The speaker suggests replacing Inverse Document Frequency (IDF) with "Information Gain". Say we have a corpus of documents each ...
user avatar
  • 205
2 votes
0 answers
67 views

Why do we use cross entropy instead of Kullback-Leibler divergence as loss function? Why do we use forward KL divergence and not the reverse?

Was just having a discussion with a colleague, and realize I have the following questions about cross-entropy that is typically used in classification problems. We know that cross entropy contains ...
user avatar
  • 445
0 votes
1 answer
48 views

Why use KL divergence if the true distribution is never known?

The KL divergence for Bayesian is information gain or loss when you sample from some distribution rather than another, where one of them is the true distribution. The problem is you never know the ...
user avatar
  • 224
1 vote
0 answers
105 views

KL divergence estimates over binary classification data

I have a dataset $D = (x_i, y_i)_{i=1}^n$ where $x_i\in \Bbb R^d$ and $y_i\in\{0, 1\}$. Suppose that $y\sim\mathrm{Bernoulli}(p(x))$ for some probability function $p:\Bbb R^d \to [0,1]$ and I would ...
user avatar
  • 343
0 votes
0 answers
15 views

KL diveregence of subsequences

Let us have a random sequence $(X_1, Y_1,\ldots,X_n,Y_n)$, where $X_t$ takes value in some set $\mathcal{X}$ and $Y_i$ are scalars. The sequence is generated by the following process: $X_i$ is chosen ...
user avatar
  • 181
1 vote
0 answers
20 views

The Role Variational Free Energy or Expected Lower Bound (ELBO) Plays as a Loss Function

In reference to variational free energy or expected lower bound, I found this sentence, "As one can easily see, the cost function tries to balance the complexity of the data P(D | w) and the ...
user avatar
0 votes
0 answers
26 views

Jensen-Shannon Distance between two Stratified Sampled Tabular Datasets

I have 100 unique joint probability mass functions with a dataset noting the prevalence of instances from each joint pmf, like this: The total amount of instances in this case would be 16,073. Each ...
user avatar
0 votes
0 answers
25 views

Measure for evaluating a density estimation procedure

Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure? I am currently evaluating the procedure using three ...
user avatar
  • 964
0 votes
0 answers
29 views

Estimate KL divergence between an unknown distribution and a known one

I am looking for a method to estimate KL divergence between a set of samples (presumably obtained from a continuous distribution) and a known, explicit distribution. I could find some algorithms for ...
user avatar
3 votes
0 answers
120 views

Label smoothing and KL divergence

I am reading the paper Regularizing Neural Networks by Penalizing Confident Output Distributions where the authors introduce label smoothing in section 3.2. For a neural network that produces a ...
user avatar
  • 436
0 votes
2 answers
463 views

Can the KL - Divergence be used to compare two time-series?

The question is pretty self-explanatory. I'm trying to compare the similarity of two time-series, in terms of the distribution of their values. I've already performed a qualitative assessment by ...
user avatar
1 vote
1 answer
60 views

$\sqrt{2 KL(f || g)}$ interpretation?

I have seen in some papers that instead of using the Kullback-Leibler divergence $KL(f || g)$ between two probability density functions, $f$ and $g$, they use $$\sqrt{2 KL(f || g)}.$$ Is there any ...
user avatar
  • 11
2 votes
0 answers
105 views

Maximum likelihood estimator and KL divergence

Let $\mathbf{X}$ be a continuous random vector, $\mathbf{d}$ a sample of size $m$, and $\mathbb{P}_{\mathbf{X}|\mathbf{\theta}}$ a parametric model for the distribution of $\mathbf{X}$. We can write ...
user avatar
  • 21
1 vote
1 answer
80 views

Kullback-Leibler distance calculation for discrete distributions?

I have the following model $$N \sim Pois(\lambda) \\ n \sim Bin(N,p)$$ for which I calculate the posterior for the parameter $N$ as $$\pi(N|n,p,\lambda) = \frac{f(n|N,p)\pi(N|\lambda)}{f(n|p,\lambda)}...
user avatar
  • 1,679
1 vote
1 answer
185 views

How to compute KL-divergence when there are categories of zero counts?

I have two very large discrete frequency distributions (about 4 million items), and each contains many items with counts of 0. I want to calculate the KL divergence between them and use the empirical ...
user avatar
1 vote
2 answers
439 views

2nd order Taylor expansion of KL divergence

I am having trouble understanding page 6 of this PDF: https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/lecture-notes/lecture20-2pp.pdf This is also a question in cs229-autumn2018 PS#3-3d. In ...
user avatar

1
2 3 4 5
10