an asymmetric measure of distance (or dissimilarity) between probability distributions.

learn more… | top users | synonyms

1
vote
1answer
29 views

EM and Kullback-Leibler divergence

Let $f$ be a density on $\mathbb{R}^{p}$. Let $f_{\theta} = \sum_{i=1}^{d} \alpha_{i}\mathcal{N}_{p}(\cdot \, ; \, \theta_{i})$ be a mixture of $d$ Gaussian distributions on $\mathbb{R}^{p}$. For each ...
0
votes
0answers
20 views

KL-divergence between two products

Given factorizations of two joint densities $p(x_1,...,x_n)=\prod_{i=1}^n p(x_i\mid \textrm{cond}(x_i))$ and $q(x_1,...,x_n)=\prod_{i=1}^n q(x_i\mid \textrm{cond}(x_i))$, where ...
4
votes
1answer
29 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
10
votes
4answers
476 views

Intuition on the Kullback-Leibler (KL) Divergence

I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say ...
3
votes
1answer
60 views

KL divergence and expectations

I am trying to understand the explanation of the KL divergence per below. It refers, as i understand it, to an expectation in the second term. "Approximating the expectation over q in this term". ...
8
votes
0answers
121 views

Special probability distribution

If $p(x)$ is a probability distribution with non-zero values on $[0,+\infty)$, for what type(s) of $p(x)$ there exist a constant $c>0$ such that $\int_0^{\infty}p(x)\log{\frac{ ...
4
votes
2answers
73 views

Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric. I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or ...
3
votes
1answer
31 views

Does the local triangle inequality holds for Kullback-Leibler divergence

Does the local triangle inequality holds for the Kullback-Leibler divergence? For the local triangle inequality, I mean the $$ d(\theta', \theta) + d(\theta'', \theta) \geq A d(\theta', \theta'') $$ ...
0
votes
0answers
48 views

Can you write a probability based on the relative entropy?

Suppose we have a graphical model $X\rightarrow \Theta \rightarrow D$ where all the distributions are Gaussian Mixture Models. Suppose further that the distribution of $X$ has more components than the ...
1
vote
0answers
33 views

Kullback-Leibler Divergence for Graph Sampling

I am from Computer Science background and need to apply Kullback Leibler Divergence to find the divergence between two distributions of unknown types. Let's say I have a graph G(V,E) and I make a ...
0
votes
0answers
30 views

Negative KL Divergence Values

I am computing KL divergence between two documents. I have got the tf-idf vectors for the top 5 features as follows: ...
1
vote
0answers
58 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
5
votes
1answer
72 views

Algorithm for approximating a density by a mixture density

Given a density $f(x)$ (e.g. the log-normal distribution or log-$t_{\nu=3}$ distribution), I was wondering what algorithm are known/typically used to find a mixture of distributions $g_r(x)$ from ...
2
votes
0answers
22 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
0
votes
1answer
26 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
4
votes
1answer
68 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
1
vote
1answer
71 views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be ...
0
votes
0answers
38 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
0
votes
0answers
17 views

Kullback-Liebler's divergence on a conditioned function

Let $q$ be a conditioned pdf over $\mathbf{X}=X_1,\dots,X_n$ binary r.v.s in the form $$q(\mathbf{X})=\begin{cases}q_{0}(\mathbf{X}_{\setminus i}) \text{ if } X_{i}=0\\q_{1}(\mathbf{X}_{\setminus i}) ...
0
votes
0answers
78 views

Minimizing KL Divergence With An Unintegrable Function in Expectation Propagation

I am trying to match the mean and variance of my posterior by minimizing the KL divergence as per the EP algorithm. However, my likelihood function is of the form:$\exp(\exp(-||\theta - x||))$ where ...
3
votes
2answers
137 views

software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's ...
1
vote
0answers
40 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
3
votes
1answer
254 views

KL divergence between two univariate Poisson distributions

I found this awesome thread which shows KL divergence between two univariate Gaussians. I was wondering if the same formula worked for KL divergence b/w 2 univariate Poisson distributions. Or should ...
2
votes
0answers
29 views

Distribution with fixed mean and closest to a given distribution

I was wondering if this problem has been tackled in some way in the probability/functional analysis literature: Given a pdf $f$ such that the expectation is zero and $\mu\in\mathbb R$, find the ...
0
votes
0answers
25 views

Trying to find a way to compare the “true” distribution of text in real life to a collection of computer generated text

I have a program that creates images of words for the purpose of training neural architectures for classifying text in image processing. The images are rendered with a number of different factors ...
3
votes
0answers
32 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. ...
0
votes
0answers
42 views

R “Entropy” package gives weird KL divergence results

Using the R "entropy" package, I tried some KL divergence computations as a sanity check, but I'm getting weird results. For instance, shouldn't the following all be 2*log2(2)= 2 ? Instead, I'm ...
2
votes
0answers
45 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
1
vote
1answer
60 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) $f(x,y) = ...
0
votes
1answer
165 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
1
vote
0answers
122 views

When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are differnt and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
0
votes
0answers
42 views

Type I errors on Hypothesis testing with KL divergence

I am performing a hypothesis test for data from an empirical distribution, $q$, where my null hypothesis is that the data is sampled from distribution $p_0$ and the alternative is that it is sampled ...
2
votes
0answers
22 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
3
votes
0answers
59 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler dinstance, so we have: $KL(p,q) = \sum_{s \in S} p(s) ...
4
votes
1answer
194 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence? $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
2
votes
1answer
141 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
2
votes
1answer
83 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = ...
1
vote
0answers
62 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
2
votes
2answers
189 views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
3
votes
1answer
461 views

Jensen Shannon Divergence vs Kullback-Leibler Divergence?

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric? Are there ...
1
vote
0answers
56 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
0
votes
3answers
379 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
0answers
66 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
1
vote
1answer
300 views
1
vote
0answers
376 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
0answers
242 views

How to estimate similarity between several probability distributions?

I have several set of probability distributions. I want to reliably estimate consistency across distributions inside each set. Literature contains methods to compare two distributions: ...
1
vote
0answers
68 views

sufficient statistic and KL-divergence: Confusion with an equation

I am reading a paper, which talks about minimising KL-divergence of any arbitrary distribution over a family of exponential distribution. So, given a distribution $p$, we want to compute its ...
1
vote
0answers
140 views

computing KL divergence: M projections for arbitrary distributions

Background I have a generative model for a process that can be described as follows: $$ y = t(x, w) + e $$ where $x$ and $y$ observations of a set of random variables which are related by a ...
0
votes
0answers
71 views

Kullback-Leibler divergence Pareto Distribution

What is the Kullback-Leibler divergence for a Pareto Distribution? Given $p(x)$ = $ \alpha$ $\frac{x^{\alpha}_{min,1}}{x^{a+1}}$.
9
votes
3answers
6k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + ...