Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

302 questions
Filter by
Sorted by
Tagged with
188 views

Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...
1k views

I'm working on a calibration problem which involves the usage of the Kullback-Leibler divergence as an error between some empirical distribution $p$ and a theoretical distribution $q$. In the model, ...
206 views

Does relative Kullback-Leibler divergence exist?

Suppose I have two multivariate normal distributions. I have computed the KL divergence ($d_{KL}(N_1, N_2)$). Is there a way to measure a relative divergence between these two distributions? For ...
3k views

KL-divergence between two categorical/multinomial distributions gives negative values?

If $$P = [0,0.9,0,0.1]$$ $$Q = [0,1,0,0]$$ Then $$KL(P||Q) = 0 + \ln(0.9/1)\cdot0.9 + 0 + 0 = -0.094$$ This shouldn't be possible from the Gibbs inequality. What am I misunderstanding?
3k views

Distance between two Gaussian mixtures to evaluate cluster solutions

I'm running a quick simulation to compare different clustering methods, and currently hit a snag trying to evaluate the cluster solutions. I know of various validation metrics (many found in cluster....
1k views

G-test statistic and KL divergence

According to Wikipedia, the G-test statistic is "proportional to the Kullback–Leibler divergence of the empirical distribution from the theoretical distribution." To get the relationship between $G$ ...
5k views

Is it possible to apply KL divergence between discrete and continuous distribution?

I am not a mathematician. I have searched the internet about KL Divergence. What I learned is the the KL divergence measures the information lost when we approximate distribution of a model with ...
1k views

2k views

Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab

Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values. How do I ...
968 views

Kullback-Leibler vs Hellinger Distance

I am working on this problem in which I have a dataset of $n$-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find $k$ examples from the dataset ...
175 views

Estimating parameters using Kullback-Leibler or Kolmogorov-Smirnoff via Nelder-Mead

I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.) Given a set of parameters $\theta$,...
1k views

Logistic regression, loss function and KL divergence

In decision theory, a loss function signature is supposed to be output space * output space -> error There seems to be many different definition of 'the ...
453 views

On the uniform convergence of relative frequencies of events to their probabilities

I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971. ...
16k views

Kullback-Leibler divergence: negative values? [duplicate]

Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g. Collection A: - word count: 321 doc count:...
1k views

5k views

Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
471 views

Similarity / dissimilarity of two large bimodal datasets

I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
23k views

Measures of similarity or distance between two covariance matrices

Are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimensions)? I am thinking here of analogues to KL divergence of two probability ...
5k views

How to compute the Kullback-Leibler divergence when the PMF contains 0s?

I have the following timeseries obtained using the data posted below. For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
343 views

How do I determine how well a dataset approximates a distribution?

Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
558 views

How to to calculate the topic distribution of a document [closed]

I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
4k views

Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and \$\...
8k views

Kullback–Leibler vs Kolmogorov-Smirnov distance

I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures. However, both are used to measure the distance between distributions. Is there a ...
77k views

KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p). ...
5k views

An adaptation of the Kullback-Leibler distance?

Look at this picture: If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
4k views

Estimate the Kullback-Leibler divergence

I would like to be sure I am able to compute the KL divergence based on a sample. Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85. set.seed(937) theta <- ....