The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
12
votes
1answer
260 views

Special probability distribution

If $p(x)$ is a probability distribution with non-zero values on $[0,+\infty)$, for what type(s) of $p(x)$ does there exist a constant $c\gt 0$ such that $\int_0^{\infty}p(x)\log{\frac{ p(x)}{(1+\...
6
votes
2answers
4k views

Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric. I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q,...
4
votes
1answer
697 views

Does the local triangle inequality holds for Kullback-Leibler divergence

Does the local triangle inequality holds for the Kullback-Leibler divergence? For the local triangle inequality, I mean the $$ d(\theta', \theta) + d(\theta'', \theta) \geq A d(\theta', \theta'') $$ ...
1
vote
0answers
71 views

Can you write a probability based on the relative entropy?

Suppose we have a graphical model $X\rightarrow \Theta \rightarrow D$ where all the distributions are Gaussian Mixture Models. Suppose further that the distribution of $X$ has more components than the ...
3
votes
1answer
557 views

Kullback-Leibler Divergence for Graph Sampling

I am from Computer Science background and need to apply Kullback Leibler Divergence to find the divergence between two distributions of unknown types. Let's say I have a graph G(V,E) and I make a ...
0
votes
0answers
640 views

What is the Kullback–Leibler divergence result for this example

Suppose we have a set of tags t1 to t6 as below, also Ua and ...
3
votes
0answers
89 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
3
votes
3answers
697 views

Quantify Difference/Distance between Lognormal distributions

I am trying to determine a metric that quantifies the distance between two continuous lognormal distributions. The data is actually a mixture of two lognormal distributions (I am not sure if this can ...
5
votes
3answers
166 views

Algorithm for approximating a density by a mixture density

Given a density $f(x)$ (e.g. the log-normal distribution or log-$t_{\nu=3}$ distribution), I was wondering what algorithm are known/typically used to find a mixture of distributions $g_r(x)$ from ...
3
votes
0answers
83 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
0
votes
1answer
44 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
5
votes
1answer
560 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
6
votes
1answer
2k views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be zero-...
3
votes
0answers
407 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
2
votes
0answers
50 views

Kullback-Liebler's divergence on a conditioned function

Let $q$ be a conditioned pdf over $\mathbf{X}=X_1,\dots,X_n$ binary r.v.s in the form $$q(\mathbf{X})=\begin{cases}q_{0}(\mathbf{X}_{\setminus i}) \text{ if } X_{i}=0\\q_{1}(\mathbf{X}_{\setminus i}) \...
4
votes
2answers
795 views

software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's ...
1
vote
0answers
107 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
4
votes
1answer
3k views

KL divergence between two univariate Poisson distributions

I found this awesome thread which shows KL divergence between two univariate Gaussians. I was wondering if the same formula worked for KL divergence b/w 2 univariate Poisson distributions. Or should ...
2
votes
0answers
50 views

Distribution with fixed mean and closest to a given distribution

I was wondering if this problem has been tackled in some way in the probability/functional analysis literature: Given a pdf $f$ such that the expectation is zero and $\mu\in\mathbb R$, find the ...
4
votes
0answers
43 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. $$...
2
votes
0answers
454 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
8
votes
2answers
595 views

Why is there a E in the name EM algorithm?

I understand where the E step happens in the algorithm (as explicated in the math section below). In my mind, the key ingenuity of the algorithm is the use of the Jensen's inequality to create a lower ...
4
votes
2answers
934 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
5
votes
1answer
1k views

When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are different and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
2
votes
0answers
36 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
5
votes
0answers
222 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler distance, so we have: $$ KL(p,q) = \sum_{s \in S} p(s) \log\left( \...
6
votes
1answer
655 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence? $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
3
votes
1answer
611 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
33
votes
2answers
10k views

Differences between Bhattacharyya distance and KL divergence

I'm looking for an intuitive explanation for the following questions: In statistics and information theory, what's the difference between Bhattacharyya distance and KL divergence, as measures of the ...
3
votes
1answer
117 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = \...
2
votes
0answers
272 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
6
votes
2answers
1k views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
11
votes
2answers
4k views

Jensen Shannon Divergence vs Kullback-Leibler Divergence?

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric? Are there ...
2
votes
0answers
82 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
16
votes
3answers
14k views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
0answers
96 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
1
vote
1answer
1k views

KLDIV Kullback-Leibler or Jensen-Shannon divergence between two distributions

let consider following probability distributions ...
3
votes
0answers
2k views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
0answers
495 views

How to estimate similarity between several probability distributions?

I have several set of probability distributions. I want to reliably estimate consistency across distributions inside each set. Literature contains methods to compare two distributions: Kullback-...
0
votes
0answers
457 views

sufficient statistic and KL-divergence: Confusion with an equation

I am reading a paper, which talks about minimising KL-divergence of any arbitrary distribution over a family of exponential distribution. So, given a distribution $p$, we want to compute its ...
1
vote
0answers
514 views

computing KL divergence: M projections for arbitrary distributions

Background I have a generative model for a process that can be described as follows: $$ y = t(x, w) + e $$ where $x$ and $y$ observations of a set of random variables which are related by a non-...
14
votes
3answers
11k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + \...
4
votes
1answer
532 views

KL divergence between an uninformative (?) Gaussian and a Gaussian

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim \mathcal{...
2
votes
0answers
495 views

Problem with Kullback–Leibler divergence criteria

I am using Kullback–Leibler divergence criteria for comparing my estimation and true density functions, but I have zero value on my estimation function when I have a testing set of size 10000, mostly ...
21
votes
2answers
6k views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
4
votes
0answers
213 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
2
votes
1answer
786 views

How to test whether a series data follow Ornstein-Uhlenbeck process (OU process)?

I have some measures which seems to have some mean-reverting properties and I'm wondering whether they can be modeled as Ornstein-Uhlenbeck process (OU process). And actually I quite expect it because ...
1
vote
1answer
216 views

How to use Kullback–Leibler divergence to help me choose the best distribution function?

I am now using Exponential distribution to model the the time intervals of a sequence of random events.Since I can choose several different lamdas for this model , I want to find out which lamda of ...
2
votes
1answer
174 views

KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, ...
1
vote
0answers
44 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...