Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
1
vote
2answers
4k views

Bhattacharyya distance for histograms

One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance. In computer vision, for example, it is used to evaluate the degree of similarity ...
5
votes
0answers
117 views

Orthogonal intersection in a Riemannian manifold

Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where $...
4
votes
1answer
369 views

Orthogonal intersection of linear family and exponential family

I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question. In statistical maniolds $S=\{p_\theta\}$,$\theta=(\theta_1,\...
20
votes
3answers
6k views

Connection between Fisher metric and the relative entropy

Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way? $$D( p(\cdot , a+da) \parallel p(\cdot,...
2
votes
1answer
2k views

Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
2
votes
0answers
891 views

Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of $n$-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find $k$ examples from the dataset ...
5
votes
0answers
174 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
1
vote
0answers
1k views

Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
2
votes
0answers
414 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
6
votes
1answer
15k views

Kullback-Leibler divergence: negative values? [duplicate]

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc count:...
3
votes
0answers
1k views

KL divergence between 2 distributions with unequal cardinalities?

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$. Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{...
3
votes
0answers
218 views

Sample distribution for Kullback-Leibler distance

For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$ and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by $$KL=...
5
votes
0answers
116 views

Lomax distributions - Kullback Leibler divergence

Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
6
votes
1answer
373 views

Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula model)...
5
votes
2answers
573 views

Maximum entropy sampler

I want to sample from a distribution which has fixed to a given values mean(=0), standard deviation(=1), skewness(=0) and kurtosis. I also want this distribution to be as general as possible, i.e. to ...
5
votes
1answer
1k views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
4
votes
0answers
217 views

Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$ r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]], $$ where $L$ is some loss function and $\hat x$ is the ...
6
votes
1answer
2k views

Kullback-Leibler divergence

Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal $p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler ...
3
votes
1answer
211 views

Property of KL-divergence

Let $p_1$ and $p_2$ be two distinct probability distributions. Define $$ L(q)=D(q||p_1)-D(q||p_2) $$ where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in ...
6
votes
1answer
948 views

Kullback–Leibler divergence between two Wishart distributions

The result is shown in: [1] W.D. Penny, KL-Divergences of Normal, Gamma, Dirichlet, and Wishart densities, Available at: www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps But could anyone help ...
1
vote
1answer
170 views

How does one express the decrease in minimal type II error bound for each observation added?

Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions: $P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian $\...
10
votes
2answers
5k views

Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
3
votes
2answers
467 views

Similarity / dissimilarity of two large bimodal datasets

I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
28
votes
4answers
22k views

Measures of similarity or distance between two covariance matrices

Are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimensions)? I am thinking here of analogues to KL divergence of two probability ...
7
votes
3answers
5k views

How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
6
votes
1answer
341 views

How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
1
vote
0answers
543 views

How to to calculate the topic distribution of a document [closed]

I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
14
votes
2answers
4k views

Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and $\...
35
votes
2answers
7k views

Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions. Is there a ...
76
votes
2answers
71k views

KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p). ...
28
votes
4answers
5k views

An adaptation of the Kullback-Leibler distance?

Look at this picture: If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
6
votes
1answer
4k views

Estimate the Kullback-Leibler divergence

I would like to be sure I am able to compute the KL divergence based on a sample. Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. set.seed(937) theta <- ....
17
votes
2answers
9k views

Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
3
votes
1answer
366 views

Properties of Battacharyya distance vs Kullback-Leibler divergence

What properties do these measures have and how can I determine which one is better for a given purpose? What are extreme cases where they differ a lot?
3
votes
0answers
96 views

What's good about I-projections?

There seems to be a large body of applied research where distribution q is picked to minimize KL(q,p) where p is empirical distribution. Are there theoretical reasons to prefer this estimator? For ...
14
votes
4answers
8k views

Questions about KL divergence?

I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...