an asymmetric measure of distance (or dissimilarity) between probability distributions.

learn more… | top users | synonyms

5
votes
0answers
42 views

Algorithm for approximating a density by a mixture density

Given a density $f(x)$ (e.g. the log-normal distribution or log-$t_{\nu=3}$ distribution), I was wondering what algorithm are known/typically used to find a mixture of distributions $g_r(x)$ from ...
2
votes
0answers
17 views

Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
0
votes
1answer
19 views

KL divergence and probabilities of 0 for P(i)

Why do probabilities of 0 for $P(i)$ not affect the result of the KL Divergence equation? Regardless of what probabilities we have for $Q(i)$, the product is 0. What are the benefits of this? Is ...
4
votes
1answer
50 views

Hypothesis test based on entropy

I am reading the wikipedia page on hypothesis testing, but a I can't find any reference to tests based on entropy. Which are good hypothesis tests based on entropy or quantities derived from it?
1
vote
1answer
43 views

variational inference with KL

i am self-studying variational inference - and in Murphy's book "A probabilistic perspective on machine learning" it is discussed that minimizing the forward KL divergence (which is stated to be ...
0
votes
0answers
19 views

How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
0
votes
0answers
13 views

Kullback-Liebler's divergence on a conditioned function

Let $q$ be a conditioned pdf over $\mathbf{X}=X_1,\dots,X_n$ binary r.v.s in the form $$q(\mathbf{X})=\begin{cases}q_{0}(\mathbf{X}_{\setminus i}) \text{ if } X_{i}=0\\q_{1}(\mathbf{X}_{\setminus i}) ...
0
votes
0answers
39 views

Minimizing KL Divergence With An Unintegrable Function in Expectation Propagation

I am trying to match the mean and variance of my posterior by minimizing the KL divergence as per the EP algorithm. However, my likelihood function is of the form:$\exp(\exp(-||\theta - x||))$ where ...
3
votes
2answers
73 views

software library to compute KL divergence?

Are there any software libraries that compute KL divergences in closed form, that also give the derivatives of the KL divergence wrt the distributions' parameters? I'm using Julia, so it's ...
1
vote
0answers
32 views

Expectation Propagation - Computing mean and variance of error function

I'm still trying to wrap my head around computing the moments for the expectation propagation algorithm and whether I can use it for the following example: say i have a product of distributions which ...
3
votes
1answer
129 views

KL divergence between two univariate Poisson distributions

I found this awesome thread which shows KL divergence between two univariate Gaussians. I was wondering if the same formula worked for KL divergence b/w 2 univariate Poisson distributions. Or should ...
2
votes
0answers
24 views

Distribution with fixed mean and closest to a given distribution

I was wondering if this problem has been tackled in some way in the probability/functional analysis literature: Given a pdf $f$ such that the expectation is zero and $\mu\in\mathbb R$, find the ...
0
votes
0answers
21 views

Trying to find a way to compare the “true” distribution of text in real life to a collection of computer generated text

I have a program that creates images of words for the purpose of training neural architectures for classifying text in image processing. The images are rendered with a number of different factors ...
3
votes
0answers
31 views

Expected ratio of probabilities--is there a term for it?

I recently came across the following quantity when I played around with some information theoretic quantities and Bayesian learning. Given three probability distributions $q(z), p(z)$ and $p(z|x)$. ...
0
votes
0answers
32 views

R “Entropy” package gives weird KL divergence results

Using the R "entropy" package, I tried some KL divergence computations as a sanity check, but I'm getting weird results. For instance, shouldn't the following all be 2*log2(2)= 2 ? Instead, I'm ...
2
votes
0answers
36 views

KL-divergence as a negative log likelihood for exponential families

I am reading Distributed Estimation, Information Loss and Exponential Families, where the authors consider and compare two estimators for $\theta$ in the parametric model $p(x\mid\theta)$: the ...
1
vote
1answer
54 views

f(y | x) or f(y,x) in regression and MLE

In $Y = aX + b + \epsilon$ where $\epsilon$ ~ $N(0,\sigma^2)$ and i.i.d regression setting If X is stochastic and $E(\epsilon\mid X) =0$, then which one is correct: (1) $f(x,y) = ...
0
votes
1answer
94 views

Relationship between Poisson generation and generalized Kullback-Leibler divergence

I have read that, in the context of matrix factorization, performing maximum likelihood estimation under the assumption that the entries are Poisson generated is equivalent to minimizing the ...
1
vote
0answers
64 views

When KL Divergence and KS test will show inconsistent results?

I know that Kullback–Leibler divergence and Kolmogorov–Smirnov test are differnt and should be used in different scenarios. But they are similar in many ways and given two distributions, we could ...
0
votes
0answers
32 views

Type I errors on Hypothesis testing with KL divergence

I am performing a hypothesis test for data from an empirical distribution, $q$, where my null hypothesis is that the data is sampled from distribution $p_0$ and the alternative is that it is sampled ...
2
votes
0answers
19 views

Approximation of objective based on statistical distance

I am a computer science researcher (mostly theoretical) currently in midst of statistics and not able to figure out how to proceed. At an abstract level, I have a hypothesis for an unknown ...
3
votes
0answers
48 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler dinstance, so we have: $KL(p,q) = \sum_{s \in S} p(s) ...
4
votes
1answer
103 views

KL divergence between a gamma distribution and a lognormal distribution?

Is there a closed-form formula for the following KL divergence? $D_{KL}(X,Y)$ where $X \sim \mathrm{Gamma}(k,\theta)$ and $Y \sim \mathrm{LogNormal}(\mu,\sigma^2)$
0
votes
0answers
14 views

How good is 0.01 Jensen-Shannon divergence score?

I want to compare a density function $p$ with the ground truth $g$ by Jensen-Shannon divergence. I am a bit unclear of how to interpret the results. In particular, how do people often conclude $p$ ...
2
votes
1answer
124 views

Minimizing KL divergence from a given distribution, according to a graph

Given $n$ discrete random variables $X_1,...,X_n$, a distribution $p$ on $X=(X_1,...,X_d)$ and a DAG (Directed Acyclic Graph) $G$ on $\{1,...,d\}$, which is the distribution $q$ factorizing with $G$ ...
2
votes
1answer
78 views

Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities. Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write: $p(x) = ...
1
vote
0answers
41 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
0
votes
0answers
28 views

Want to findout satistical distance unsing another procedure except kullback liebler divergence

i want to find the distance between two pdf(pdf is calculated using kernel density estimator from two random data set of different size ) .Is there any alternative and efficent way to calculate ...
2
votes
2answers
132 views

Why KL-Divergence uses “ln” in its formula?

I notice in KL-Divergence formula a $ln$ function is used: $${D_{KL}}(P||Q) = \sum\limits_i {P(i)} \ln \frac{{P(i)}}{{Q(i)}},$$ where $i$ is a point and $P(i)$ the true discrete probability ...
3
votes
1answer
305 views

Jensen Shannon Divergence vs Kullback-Leibler Divergence?

I know that KL Divergence is not symmetric and it cannot be strictly considered as a metric. If so, why is it used when JS Divergence satisfies the required properties for a metric? Are there ...
0
votes
0answers
35 views

lower bound on square root of KL divergence

$\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and $f_2$. I wonder if the following is a correct usage of Jensen' inequality: $$ ...
0
votes
0answers
53 views

Difference between two data distributions

I have the following problem: I have two datasets: A and B. From each of the datasets, I extracted the same features. I also know the data labels. So, I would like to perform 10-fold CV to see how a ...
1
vote
0answers
38 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
0
votes
3answers
231 views

Analysis of Kullback-Leibler divergence

Let us consider the following two probability distributions ...
1
vote
0answers
58 views

Neuroscience Equations

I am trying to understand a neuroscience article by Karl Friston. In it he gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and ...
1
vote
1answer
252 views
1
vote
0answers
246 views

Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
1
vote
0answers
63 views

How to estimate similarity between several probability distributions?

I have several set of probability distributions. I want to reliably estimate consistency across distributions inside each set. Literature contains methods to compare two distributions: ...
1
vote
0answers
60 views

sufficient statistic and KL-divergence: Confusion with an equation

I am reading a paper, which talks about minimising KL-divergence of any arbitrary distribution over a family of exponential distribution. So, given a distribution $p$, we want to compute its ...
1
vote
0answers
100 views

computing KL divergence: M projections for arbitrary distributions

Background I have a generative model for a process that can be described as follows: $$ y = t(x, w) + e $$ where $x$ and $y$ observations of a set of random variables which are related by a ...
0
votes
0answers
58 views

Kullback-Leibler divergence Pareto Distribution

What is the Kullback-Leibler divergence for a Pareto Distribution? Given $p(x)$ = $ \alpha$ $\frac{x^{\alpha}_{min,1}}{x^{a+1}}$.
9
votes
3answers
5k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + ...
3
votes
1answer
153 views

KL divergence between an uninformative (?) Gaussian and a Gaussian

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim ...
1
vote
0answers
132 views

Problem with Kullback–Leibler divergence criteria

I am using Kullback–Leibler divergence criteria for comparing my estimation and true density functions, but I have zero value on my estimation function when I have a testing set of size 10000, mostly ...
11
votes
2answers
863 views

What is the relationship between the GINI score and the log-likelihood ratio

I am studying classification and regression trees, and one of the measures for the split location is the GINI score. Now I am used to determining best split location when the log of the likelihood ...
2
votes
0answers
85 views

Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
2
votes
1answer
226 views

How to test whether a series data follow Ornstein-Uhlenbeck process (OU process)?

I have some measures which seems to have some mean-reverting properties and I'm wondering whether they can be modeled as Ornstein-Uhlenbeck process (OU process). And actually I quite expect it because ...
0
votes
1answer
46 views

How to use Kullback–Leibler divergence to help me choose the best distribution function?

I am now using Exponential distribution to model the the time intervals of a sequence of random events.Since I can choose several different lamdas for this model , I want to find out which lamda of ...
1
vote
1answer
67 views

KL divergence minimisation equation

I am looking at some literature on KL divergence minimisation and am having trouble understanding the derivation of the second order moment. So, if we have a distribution from the exponential family, ...
1
vote
0answers
26 views

Modeling a list with a tunable degree of disorder/shuffling

Imagine we have a list of ordered numbers $L = (1, 2,\dots, N)$. I want to add an arbitrary amount of "disorder" to that list. For instance: Adding a little bit of disorder would permute a few ...