an asymmetric measure of distance (or dissimilarity) between probability distributions.

learn more… | top users | synonyms

0
votes
0answers
15 views

Why when minimizing the forward Kullback-Leibler Divergence w.r.t a factored approximation is it correct to set Q to be the product of marginals?

The following question is from Mackay's 2004 textbook. Consider the problem of approximating a joint distribution $P(x, y)$ by a separable distribution $Q(x, y) = Q_X(x)Q_Y(y)$. Show that if the ...
0
votes
0answers
10 views

Approximating KL-Divergence for 2-D Random Variables with Scatter Plots

I have lots of experience computing KL divergences for straightforward discrete distributions where I have access to complete probability tables, etc. But I'm a little concerned about my current ...
1
vote
1answer
103 views

Monotonicity of special case of Kullback-Leibler divergence

I have two discrete distributions $\tau$ and $\rho$ with the same support $\Omega$. I'm considering a weighted mixture of these distributions described by the following function: $$ f(w) = (1-w) \cdot ...
0
votes
0answers
47 views

Kullback-Leibler Divergence

I tried to implement a numerical estimate of the Kullback-Leibler Divergence for two samples. To debug the implementation draw the samples from two normal distributions $\mathcal N (0,1)$ and ...
0
votes
0answers
15 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
0
votes
0answers
14 views

KL-Information with n=1 vs n=100

I am reading about the Kullback-Leibler Information in Model Selection and Multimodel Inference 2nd ed. by Burnham and Anderson [1]. For those who do not know, KL-Information is defined as: ...
3
votes
2answers
59 views

What are the assumptions that we make when we compute KL Divergence between two distributions?

Let us assume that we compute the KL Divergence between p and q. Is it necessary that both p and q belong to the exponential family of distributions. Moreover, is it necessary, that both p and q ...
1
vote
0answers
20 views

What tests can I use to compare these two probability distributions

I am trying to compare two one-dimensional distributions. I am using Kullback-Leibler divergence function for this but it requires me to have both the distributions of equal length. I am not sure how ...
1
vote
0answers
31 views

How to compare posterior distributions for different observed data? KL-divergence?

So I'm solving an inverse problem with the Bayesian approach $p(u | y) \propto p(y| u )p(u)$. Assuming I have two datasets $y_1$ and $y_2$, what can be said about the difference in the posteriors ...
1
vote
0answers
13 views

Converting Dirichlet distribution to distribution on the log-linear parameters

Dirichlet prior/posterior provides a probability density on distributions over a multinomial variable. It has the form : $P(P) \varpropto \prod_i{P_i^{\alpha_i-1}}$ I can also describe the ...
3
votes
0answers
17 views

Properties of Average Multinomial Likelihood

I am trying to understand the Kullback-Leibler Information: I read in http://arxiv.org/pdf/1404.2000v1.pdf the following: Ideally, we want the probability to be invariant to the number of ...
0
votes
0answers
15 views

Kullback-Leibler and Battacharyya divergences between Hidden Markov Models with discrete emissions

Im trying to figure out how to compute KL or Battacharyya divergences between two HMMs models. I found papers which are about HMMs with normaly distributed emissions, but nothing for discrete ...
6
votes
1answer
69 views

$\min D_\textrm{KL}(p(x_1,\dots,x_n) \mid\mid q_1(x_1)\cdots q_n(x_n))$ gives the marginals of $p(x_1,\dots,x_n)$?

Prove or disprove: Let $p(x_1,\dots,x_n)$ be a given probability distribution over the $n$ variables $x_1, \dots,x_n$. The univariate probability distributions $q(x_1),\dots,q(x_n)$ that minimize the ...
0
votes
1answer
61 views

How to apply distance metrics to compare bar plot (nominal histogram) data

I have a data set for libraries, I would like to find the (Similarity / dissimilarity) among it based on book category, so for each category there is single value represent the number of books that ...
0
votes
0answers
18 views

How to compare 2 estimations of a probability distribution?

I am using a program that returns a probability estimation Q to predict a value. I have access to the real probability distribution P that I try to estimate, from the dataset. I can compute the ...
1
vote
1answer
53 views

EM and Kullback-Leibler divergence

Let $f$ be a density on $\mathbb{R}^{p}$. Let $f_{\theta} = \sum_{i=1}^{d} \alpha_{i}\mathcal{N}_{p}(\cdot \, ; \, \theta_{i})$ be a mixture of $d$ Gaussian distributions on $\mathbb{R}^{p}$. For each ...
0
votes
0answers
30 views

KL-divergence between two products

Given factorizations of two joint densities $p(x_1,...,x_n)=\prod_{i=1}^n p(x_i\mid \textrm{cond}(x_i))$ and $q(x_1,...,x_n)=\prod_{i=1}^n q(x_i\mid \textrm{cond}(x_i))$, where ...
4
votes
1answer
35 views

Minimize $K(p||q)$, when $q$ is not normalizable?

Let $K(p||q)$: $$K(p||q) = \int p(x) \log \frac{p(x)}{q(x)} \mathrm{d} x$$ where the integral goes over the common support of $p$ and $q$. The distribution $p$ that minimizes this is $p = q$. ...
13
votes
4answers
687 views

Intuition on the Kullback-Leibler (KL) Divergence

I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say ...
3
votes
1answer
100 views

KL divergence and expectations

I am trying to understand the explanation of the KL divergence per below. It refers, as i understand it, to an expectation in the second term. "Approximating the expectation over q in this term". ...
9
votes
0answers
147 views

Special probability distribution

If $p(x)$ is a probability distribution with non-zero values on $[0,+\infty)$, for what type(s) of $p(x)$ there exist a constant $c>0$ such that $\int_0^{\infty}p(x)\log{\frac{ ...
4
votes
2answers
108 views

Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric. I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or ...
3
votes
1answer
38 views

Does the local triangle inequality holds for Kullback-Leibler divergence

Does the local triangle inequality holds for the Kullback-Leibler divergence? For the local triangle inequality, I mean the $$ d(\theta', \theta) + d(\theta'', \theta) \geq A d(\theta', \theta'') $$ ...
0
votes
0answers
50 views

Can you write a probability based on the relative entropy?

Suppose we have a graphical model $X\rightarrow \Theta \rightarrow D$ where all the distributions are Gaussian Mixture Models. Suppose further that the distribution of $X$ has more components than the ...
1
vote
0answers
54 views

Kullback-Leibler Divergence for Graph Sampling

I am from Computer Science background and need to apply Kullback Leibler Divergence to find the divergence between two distributions of unknown types. Let's say I have a graph G(V,E) and I make a ...
0
votes
0answers
40 views

Negative KL Divergence Values

I am computing KL divergence between two documents. I have got the tf-idf vectors for the top 5 features as follows: ...
2
votes
0answers
60 views

Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
5
votes
1answer
76 views

Algorithm for approximating a density by a mixture density

Given a density $f(x)$ (e.g. the log-normal distribution or log-$t_{\nu=3}$ distribution), I was wondering what algorithm are known/typically used to find a mixture of distributions $g_r(x)$ from ...
2
votes
0answers
22 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
0
votes
1answer
27 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
4
votes
1answer
88 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
1
vote
1answer
147 views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be ...
1
vote
0answers
41 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
1
vote
0answers
23 views

Kullback-Liebler's divergence on a conditioned function

Let $q$ be a conditioned pdf over $\mathbf{X}=X_1,\dots,X_n$ binary r.v.s in the form $$q(\mathbf{X})=\begin{cases}q_{0}(\mathbf{X}_{\setminus i}) \text{ if } X_{i}=0\\q_{1}(\mathbf{X}_{\setminus i}) ...
3
votes
2answers
199 views

software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's ...
1
vote
0answers
44 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
3
votes
1answer
422 views

KL divergence between two univariate Poisson distributions

I found this awesome thread which shows KL divergence between two univariate Gaussians. I was wondering if the same formula worked for KL divergence b/w 2 univariate Poisson distributions. Or should ...
2
votes
0answers
30 views

Distribution with fixed mean and closest to a given distribution

I was wondering if this problem has been tackled in some way in the probability/functional analysis literature: Given a pdf $f$ such that the expectation is zero and $\mu\in\mathbb R$, find the ...
0
votes
0answers
25 views

Trying to find a way to compare the “true” distribution of text in real life to a collection of computer generated text

I have a program that creates images of words for the purpose of training neural architectures for classifying text in image processing. The images are rendered with a number of different factors ...
3
votes
0answers
33 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. ...
2
votes
0answers
56 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
1
vote
1answer
60 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) $f(x,y) = ...
0
votes
1answer
197 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
1
vote
0answers
177 views

When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are differnt and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
2
votes
0answers
24 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
3
votes
0answers
63 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler dinstance, so we have: $KL(p,q) = \sum_{s \in S} p(s) ...
4
votes
1answer
243 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence? $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
2
votes
1answer
161 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
2
votes
1answer
85 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = ...
1
vote
0answers
95 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...