Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Centroid wrt Kullback-Leibler divergence with Kronecker deltas

I have the following problem. Suppose I have a set of discrete probabilities $f_i:A \rightarrow[0,1], \ \ i=1,...,m$, such that $\forall i \ \ \exists a_i : f_i(a_i) =1$, that is they are constant. ...
6
votes
1answer
533 views

Is my proof that relative entropy is never negative correct?

I wish to prove that relative entropy(Kullback-Liebler divergence) is always non-negative. I.e. that $$I^{KL}(F;G)=E_F\left[\log\frac{f(X)}{g(X)}\right]\geq0$$ where F,G are two different probability ...
1
vote
2answers
30 views

When to use distance between distributions instead of using median based statistics?

I have a problem where I have to compare the effect of X ( univariate random variable) on distribution of Y (univariate random variable) between 2 different cases. Y is not following a Normal ...
0
votes
0answers
25 views

Kullback–Leibler divergence between two inverse Wishart distributions

Is there a closed form formula for the KL divergence between two inverse Wishart distributions?
1
vote
1answer
30 views

Variational autoencoders: Computational vs. analytical intractability of KL divergence

I am currently trying to understand the ideas behind variational autoencoders. Specifically, I am a trying to understand why the KL divergence between the approximate posterior $q(z | x)$ and true ...
2
votes
1answer
38 views

If the KL divergence is not a metric or a measure, what is it?

The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, ...
0
votes
1answer
29 views

KL-Divergence and Entropy for marginals

I am going through this paper, where the following claim is unproven (page 3, after the first equality): Let $r,c \in \mathbb{R}_+^d$ be discrete probability histograms, and $P \in \mathbb{R}_+^{d,d}$...
5
votes
2answers
163 views

Relationship between KL divergence and entropy

In Bishop's Pattern Recognition and Machine Learning, there is a small discussion in section 10.1.2 of the difference between minimizing $D_{KL}(p \:||\: q)$ and $D_{KL}(q \:||\: p)$ with respect to ...
4
votes
1answer
72 views

Why is Kullback Leibler Divergence always positive?

I know there have been mathematical treatments of this question on here. What I'd like help with is my intuitive understanding though. Take the example given on Wikipedia: $$\begin{array}{|c|c|c|c|} \...
2
votes
1answer
23 views

Can $G^2$ statistic in log-linear model for contingency tables be negative?

Can $G^2$ statistic of log-linear (unsaturated) model in contingency tables be negative? Since saturated model with perfect fit has $G^2=0$ I don't think the unsaturated models can get negative $G^2$. ...
0
votes
0answers
18 views

Is the KL-Divergence invariant to strictly monotonic transformations of the random variable?

Let $p$ and $q$ be two distributions on a variable $X$. Let $\widetilde{p}$ and $\widetilde{q}$ be the corresponding distributions on $f(X)$, where $f$ is a strictly monotonic function (e.g. $f(x)=e^x$...
0
votes
0answers
56 views

KL divergence for Generalized Extreme Value distribution

I have found a derivation for the Kullback–Leibler divergence between 2 Gumbel distributions here: http://www.mast.queensu.ca/~communications/Papers/gil-msc11.pdf on page 64 That document also has a ...
1
vote
1answer
56 views

Efficiently computing pairwise KL divergence between multiple diagonal-covariance Gaussian distributions

Let's say I want to compute the pairwise KL divergence between a large number (O(100)) of multivariate Gaussian distributions with diagonal covariance. The mean parameters for each Gaussian are stored ...
0
votes
0answers
28 views

Interpreting KL Divergence

I'm trying to compare different approaches to rank predictions. I have the ground truth distribution $P$ (discrete, zeta distribution) and two or more distributions ($Q, Q', Q'', Q'''$ in this case) I'...
0
votes
0answers
15 views

Can we upper bound the KL by BIC or AIC criteria?

I would like to know if it is possible to bound the KL (or the Bayesian free energy a.k.a negative log marginal likelihood) by the Bayesian (and or Akaike) information criterion. Thank you!
0
votes
0answers
6 views

KL-div regularization in policy-based RL

I have a question about using a KL divergence as a regularization term in policy-based RL. Previously, I was using an entropy bonus in my policy objective: $$ \text{objective} = \text{ppo_clip} + \...
2
votes
1answer
42 views

Additive property of KL-divergence

I am new to information theory and I want to know is there any conclusion about the relationship between $KL \left( p*a + (1-p)*b | c \right)$ and $p*KL \left( a | c \right) + (1-p) * KL \left( b | c \...
0
votes
0answers
24 views

Estimate a Mean using Monte Carlo Integration

Suppose $\hat f(x)$ is the KDE $$\hat f(x) = \frac{1}{nh}\sum_{i=1}^nK\left(\frac{x - X_i}{h}\right).$$ Now I want to estimate the KL divergence to the true density $f$ using an MC approach: $$KL = \...
0
votes
0answers
25 views

Kullback-Leibler Divergence for histograms: How to choose bin size?

So I have two set of data points. To be precise these are measurement of surface areas of some objects using two different methodologies (which could give a different object count and different areas)....
1
vote
1answer
48 views

Relationship between KL divergence and correlation

I know KL divergence tries to measure how different 2 probability distributions are. I know high correlation values between 2 sets of variables imply they are highly dependent on each other. Will ...
1
vote
0answers
18 views

KL divergence of categorical distribution with continuous inputs

I want to simulate a process. I have a probability distribution and I have d classes to choose from. The inputs of my distribution are 3d points and it maps each of these points to a d-dimensional ...
3
votes
1answer
50 views

Difference of two KL-divergence

The Kullback-Leibler (KL) divergence between two distributions $P$ and $Q$ is defined as $$\mbox{KL}(P \| Q) = \mathbb{E}_P\left[\ln \frac{\mbox{d}P}{\mbox{d}Q}\right].$$ My question is that suppose ...
2
votes
0answers
30 views

Kullback-Leibler Divergence continuous for measures?

in one of the commments to this post concerning the application of Kullback-Leibler-divergence between measures that do not fulfill the necessary absolute continuity (e.g. point mass vs. continuous) , ...
1
vote
0answers
47 views

How to add a $\beta$ and capacity term to a variational autoencoder?

Vanilla variational autoencoders add a Kullback–Leibler divergence (KL) term to the loss function, i.e. the loss is a combination of the reconstruction error (e.g. cross entropy or MSE) and the KL ...
0
votes
0answers
12 views

What cost function is used by sklearn's logistic regression in binary classification?

I need to use the LogisticRegression from scikit-learn for a binary classification task. For classification problems, I've mainly used tensorflow/keras where you specify which cost function you want ...
1
vote
0answers
23 views

Gaussian closest to a maximum of Gaussians

Let $X_i \sim \mathcal{N}(\mu_i, \sigma_i)$ be independent, normally-distributed random variables. Let $$Y = a + b \max_i X_i$$ where $a \in \mathbb{R}$ and $b \in (0, 1)$. Which Gaussian ...
4
votes
1answer
57 views

Bayesian linear regression KL divergence

$$y_i \sim N(w_0 + w_1x_i, \sigma^2_j)$$ $$\mathbf{w} \sim N(0,\alpha^2 I) $$ Data is $D$, posterior distribution $p(\mathbf{w}|D)$ is approximated according to mean-field approximation $$p(\mathbf{w}|...
0
votes
0answers
11 views

Kullback-Leibler divergence from density f to density g

If $f$ and $g$ are density functions that are positive over the same region, then the Kullback-Leibler divergence from density $f$ to density $g$ is defined by: $$KL(f,g) = E_f\left[\ln\left(\frac{f(...
0
votes
1answer
30 views

Why is the mean and log variance specified as the output of an inference network in a variational autoencoder?

This question was also asked in the link below but I don't think the answer there really addressed the question. Latent output of variational autoencoder So why the log variance and mean, why not ...
0
votes
0answers
21 views

Is there a connection between the binomial pmf and the formula for entropy? [duplicate]

Many times when two formulas "look" the same, there is some interesting mathematical result linking them. Both the log binomial likelihood and the entropy formula kind of "look" the same, in that they ...
1
vote
0answers
22 views

Does minimizing KL imply boundedness of density ratios

Suppose $q^*(\theta) = \underset{q \in \mathcal{Q}}{argmin} \,\, KL[q(\theta)||p(\theta)] = \min_{q\in\mathcal{Q}} \int_{\theta\in\Theta}q(\theta)\ln\Big( \frac{q(\theta)}{p(\theta)} \Big)d\theta$ ...
0
votes
0answers
21 views

Meaning of entropy for a multivariate data set

In an earlier question, I asked, “Any meaning to the concept of ‘Self Mutual Information?” and got a great answer - thanks. Now, this begs another question/set of questions: What does the concept of ...
3
votes
0answers
101 views

KL divergence between two multivariate Gaussians with close means and variances

KL divergence between two Gaussian distributions denoted by $\mathcal{N}(\mathbf \mu_1, \mathbf \Sigma_1)$ and $\mathcal{N}(\mathbf \mu_2, \mathbf \Sigma_2)$ is available in a closed form as: $$\...
1
vote
0answers
19 views

Kullback-Leibler Divergence vs Normalized Cross Correlation

I have couple of time series data that I want to cluster. As I was looking for ways to calculate similarity for time series data, I came across couple of different similarity methods. What I am ...
1
vote
0answers
38 views

Mathematical meaning of minimizing JS divergence about GAN [closed]

Optimization of the loss function of GAN is equivalent to minimizing Jensen Shannon divergence, and minimization of cross-entropy loss, which is often used in classification problems such as image ...
1
vote
1answer
207 views

maximizing KL divergence as the objective function

As far as I know, the most common approach to train neural networks is to minimize the KL divergence between the data distribution and the output of the model distribution which results in minimizing ...
0
votes
1answer
53 views

Derivations of Forward and Reverse KL Divergence equations

In the Forward KL, the entropy has disappeared and in the Reverse KL, the entropy has a plus sign, why are they so?
0
votes
0answers
10 views

Use boostrap to compute confidence interval of KL statistic

I have two sample distributions p and q. samples in p are maybe identically distributed to samples in q or not. this is the task. samples within one group are iid. p and q have the same cardinality. ...
0
votes
0answers
45 views

Jensen–Shannon divergence as a distance measure between nonprobabilistic objects

We are working on an optimization problem. The objective function involves distance between data points. We tried a wide variety of distance measures and found the entropy-based measures, especially ...
0
votes
0answers
32 views

Kullback-Leibler (KL) divergence cutoff value

I am performing the KL divergence method to compare distributions of variables between two groups. I have a list of variables within different categories (award types, organization types, topics, etc) ...
0
votes
1answer
72 views

Maximum likelihood and Minimizing Kullback–Leibler divergence to the ecdf?! (i.e.: finite sample statement?)

There is a known relationship stating that finding MLE is asymptotically the same is minimizing Kullback–Leibler divergence (see wiki here), or just the cross entropy. I'm wondering if there is a ...
2
votes
1answer
53 views

“Any meaning to the concept of ‘Self Mutual Information?”

** “Any meaning to the concept of ‘Self Mutual Information?” ** A blog post entitled, “Entropy in machine learning” dated May 6, 2019 (https://amethix.com/entropy-in-machine-learning/) gave a very ...
1
vote
0answers
108 views

KL divergence between samples from a unknown distribution and a Normal distribution with zero mean and unit variance

If you draw samples of unknown distribution, how can you measure the KL-divergence between the unknown distribution and a gaussian distribution with zero mean and unit variance N(0,1)? Can we use ...
0
votes
0answers
9 views

Is Kullback-Leibler distance the same as the Kullback Information Criterion?

I have seen some old papers, where they seem to be referring to Kullback-Leibler divergence as the Kullback Information Criterion (e.g. Gourieroux et al (1987) "Kullback Causality Measures"). I just ...
1
vote
0answers
45 views

Statistical estimators distance for close empirical distributions

Is it valid to argue that two empirical distributions $ p_1, p_2 $ having small Wasserstein distance $W_r(\cdot)$ for an order $ r $ will yield close MLE estimators for a statistical model ...
1
vote
1answer
123 views

Kullback Leibler Divergence between two Normal Whishart Distributions

I'm having trouble to compute the KL Divergence between two normal-Wishart distributions. KL divergence from $Q$ to $P$ is defined as: $$D_{\mathrm{KL}}(P \Vert Q) = \int p(x) \log \frac{p(x)}{q(x)}...
0
votes
0answers
43 views

Clarifying notation for the Kullback-Leibler divergence in terms of expectations

We know that for discrete variables \begin{equation} D(p(x),q(x))=\mathbb{E}_{p}\left(\log\frac{p(x)}{q(x)}\right) \end{equation} where $p(x)$ and $q(x)$ are probability mass functions. Can this be ...
1
vote
0answers
40 views

Distance measure for two probability distribution of unequal sample size

Context: I have 100 stores and these stores are divided into 10 business markets. I want to select 3 markets where each market is a good representation of the 100 stores i.e. the population. There ...
0
votes
0answers
26 views

Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
1
vote
1answer
146 views

KL divergence between two Asymmetric Laplace distributions?

Consider two asymmetric Laplace distribution $L_1(\mu_1,\sigma_1,\tau_1)$ and $L_2(\mu_2,\sigma_2,\tau_2)$ where \begin{equation} L(x;\mu,\sigma,\tau) =\frac{\tau(1-\tau)}{\sigma} \begin{cases} ...

1
2 3 4 5
8