Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

150 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8
votes
0answers
2k views

What is the KL divergence of distribution from Dirac delta?

The Kullback–Leibler (KL) divergence of two continuous distributions $P(x)$ and $Q(x)$ is defined as $$D_{KL}(P \mid\mid Q) = \int_{X} P(x) \log{\left[\frac{P(x)}{Q(x)}\right]} dx$$ How can one ...
6
votes
0answers
125 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where $...
5
votes
0answers
244 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler distance, so we have: $$ KL(p,q) = \sum_{s \in S} p(s) \log\left( \...
5
votes
0answers
180 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
5
votes
0answers
125 views

Lomax distributions - Kullback Leibler divergence

Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
4
votes
1answer
276 views

Connection between log predictive density and Kullback-Leibler information measure

I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information. The log ...
4
votes
0answers
879 views

Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
4
votes
0answers
1k views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
4
votes
0answers
102 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
4
votes
0answers
48 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. $$...
4
votes
0answers
222 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
4
votes
0answers
225 views

Sample distribution for Kullback-Leibler distance

For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$ and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by $$KL=...
4
votes
0answers
224 views

Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$ r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]], $$ where $L$ is some loss function and $\hat x$ is the ...
3
votes
0answers
13 views

What's an intuitive way to understand the bias of KL divergence?

The general intuition I have seen for KL divergence is that it computes the difference in expected length sampling from distribution P with an optimal code for P vs sampling from distribution P with ...
3
votes
1answer
60 views

Difference of two KL-divergence

The Kullback-Leibler (KL) divergence between two distributions $P$ and $Q$ is defined as $$\mbox{KL}(P \| Q) = \mathbb{E}_P\left[\ln \frac{\mbox{d}P}{\mbox{d}Q}\right].$$ My question is that suppose ...
3
votes
0answers
53 views

Kullback-Leibler Divergence continuous for measures?

in one of the commments to this post concerning the application of Kullback-Leibler-divergence between measures that do not fulfill the necessary absolute continuity (e.g. point mass vs. continuous) , ...
3
votes
0answers
146 views

KL divergence between two multivariate Gaussians with close means and variances

KL divergence between two Gaussian distributions denoted by $\mathcal{N}(\mathbf \mu_1, \mathbf \Sigma_1)$ and $\mathcal{N}(\mathbf \mu_2, \mathbf \Sigma_2)$ is available in a closed form as: $$\...
3
votes
0answers
289 views

Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
3
votes
0answers
40 views

What is the distribution G (with given mean and variance) that has the minimum KL distance from normal distribution F?

I was hoping someone knows whether the following statement is true or not. Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (...
3
votes
0answers
153 views

How to use Kullback-Leiber divergence for discriminating 2 keywords

I'm getting 500 tweets for each of the keywords K1 and K2 and I want to discriminate these 2 keywords with Kullback-Leiber Divergence formula. I do normal text mining preprocessing such as removing ...
3
votes
0answers
84 views

Properties of Average Multinomial Likelihood

I am trying to understand the Kullback-Leibler Information: I read in http://arxiv.org/pdf/1404.2000v1.pdf the following: Ideally, we want the probability to be invariant to the number of ...
3
votes
0answers
100 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
3
votes
0answers
484 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
3
votes
0answers
2k views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
3
votes
0answers
191 views

Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...
3
votes
0answers
1k views

KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{...
3
votes
0answers
149 views

Relative entropy optimization

I have what I hope should be a relatively quick and easy to answer question. Would it ever make sense to replace the probabilities by an objective function within the relative entropy formulation? I ...
3
votes
0answers
100 views

What's good about I-projections?

There seems to be a large body of applied research where distribution q is picked to minimize KL(q,p) where p is empirical distribution. Are there theoretical reasons to prefer this estimator? For ...
2
votes
1answer
19 views

what is -0.5 in VAE loss function with KL term

The VAE loss is composed of two terms: Reconstruction loss KLD loss in the implementation there is -0.5 applied to KLD loss. Kindly let me know what is this -0.5
2
votes
0answers
60 views

If the KL divergence is not a metric or a measure, what is it?

The KL divergence is not a metric because e.g. it does not satisfy the symmetry property that metrics posses. According to the definition of measure, the KL divergence doesn't seem to be a measure, ...
2
votes
0answers
163 views

KL divergence between samples from a unknown distribution and a Normal distribution with zero mean and unit variance

If you draw samples of unknown distribution, how can you measure the KL-divergence between the unknown distribution and a gaussian distribution with zero mean and unit variance N(0,1)? Can we use ...
2
votes
0answers
51 views

A divergence that can be extended to logistic functions?

If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function: $$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$ ...
2
votes
0answers
182 views

Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
2
votes
0answers
43 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
2
votes
0answers
300 views

cont. KL divergence in python

...
2
votes
0answers
209 views

Relation between the AIC and the Kullback-Leibler Divergence

I am searching a formal derivation of the Akaike Information Criterion from the Kullback-Leibler Divergence. Can you show me one, or point me toward a book/article in which this is done? Here I set ...
2
votes
0answers
402 views

If the AIC is an estimate of Kullback-Leibler Divergence, then why can AIC be negative when KL divergence is always positive?

I have read many times that the AIC serves as an estimate of the KL divergence, and I know that AIC can be a negative value (and have seen that myself). Yet, the KL divergence must always be positive. ...
2
votes
0answers
243 views

KL divergence and anomaly detection

stats newbie here. I have a dataset that is collected weekly. In order to make sure the data set gathered this week conform to past observations, I'm using KL divergence to compute how similar the ...
2
votes
0answers
55 views

Inequality Kullback divergence

I have a problem with solving the following question. Let $\mathcal{P} = \{\mathbb{P}_\theta : \theta \in \Theta\}$ be a statistical family of discrete distributions with state space $\mathcal{X}$ ...
2
votes
0answers
101 views

Interpreting base measure in exponential family as an improper prior (because entropy)

Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base ...
2
votes
0answers
424 views

Deriving the gradient of the loss in SNE

The objective used in SNE is the KL divergence between the two distributions and is given as $$ E(Y) = \sum_i \sum_j p_{j|i}\log \frac{p_{j|i}}{q_{j|i}} $$ and the two distributions are as follows, $...
2
votes
0answers
315 views

KL divergence between function of two distributions

Suppose we have two random variables: $X \sim f(x)$ and $X' \sim f'(x)$ and we know how to compute $KL(f||f')$. We apply a one to one function $g$ on both random variables which gives us two new ...
2
votes
0answers
242 views

KL divergence in Sequential Monte Carlo

Suppose at step $t$ the particle approximation of SMC in $d$ dimensions is given by $\sum_{k=1}^N w_k\delta(\vec{x}-\vec{x}_k)$, and at the subsequent step, $t+1$ (after using Bayes' law to update ...
2
votes
0answers
52 views

Comparing approximating mixture distributions

Setup: Say I have some Bayesian predictive model I assume to be true for each observation $x_1$. Each $x_2$ is a latent/unseen/hidden random variable. The parameters are $\theta$. It's a mixture ...
2
votes
0answers
54 views

Kullback-Liebler's divergence on a conditioned function

Let $q$ be a conditioned pdf over $\mathbf{X}=X_1,\dots,X_n$ binary r.v.s in the form $$q(\mathbf{X})=\begin{cases}q_{0}(\mathbf{X}_{\setminus i}) \text{ if } X_{i}=0\\q_{1}(\mathbf{X}_{\setminus i}) \...
2
votes
0answers
60 views

Distribution with fixed mean and closest to a given distribution

I was wondering if this problem has been tackled in some way in the probability/functional analysis literature: Given a pdf $f$ such that the expectation is zero and $\mu\in\mathbb R$, find the ...
2
votes
0answers
499 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
2
votes
0answers
41 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
2
votes
0answers
318 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
2
votes
0answers
82 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...