# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

296 questions
Filter by
Sorted by
Tagged with
5k views

### Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
1k views

2k views

### Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
922 views

### Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of $n$-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find $k$ examples from the dataset ...
174 views

### Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
1k views

### Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
418 views

### On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
15k views

### Kullback-Leibler divergence: negative values? [duplicate]

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc count:...
1k views

5k views

### Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
468 views

### Similarity / dissimilarity of two large bimodal datasets

I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
22k views

### Measures of similarity or distance between two covariance matrices

Are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimensions)? I am thinking here of analogues to KL divergence of two probability ...
5k views

### How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
343 views

### How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
549 views

### How to to calculate the topic distribution of a document [closed]

I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
4k views

### Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and \$\...
8k views

### Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions. Is there a ...
73k views

### KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p). ...
5k views

### An adaptation of the Kullback-Leibler distance?

Look at this picture: If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
4k views

### Estimate the Kullback-Leibler divergence

I would like to be sure I am able to compute the KL divergence based on a sample. Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. set.seed(937) theta <- ....
9k views

### Kullback-Leibler divergence - interpretation [duplicate]

I have a question about the Kullback-Leibler divergence. Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
366 views

### Properties of Battacharyya distance vs Kullback-Leibler divergence

What properties do these measures have and how can I determine which one is better for a given purpose? What are extreme cases where they differ a lot?