# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

312 questions
Filter by
Sorted by
Tagged with
136 views

### Choosing a Model Selection Criterion [closed]

I am trying to decide which among for model selection criteria to use for a Bayesian nonparametric model. The candidates are: The L-criterion, as defined by Laud & Ibrahim (1995); Bayes factors; ...
39 views

### What is the distribution G (with given mean and variance) that has the minimum KL distance from normal distribution F?

I was hoping someone knows whether the following statement is true or not. Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (...
2k views

### Estimate the Kullback Leibler (KL) divergence with monte carlo

I want to estimate the KL divergence between two continuous distributions f and g. However, I can't write down the density for either f or g. I can sample from both f and g via some method (for ...
2k views

### Kullback-Leibler distance for comparing two distribution from sample points

I have two data samples of a value and I want to compute some distance which would represent the difference in their distribution. I read about Kullback-Leibler distance which could be used for ...
135 views

### What is the probability that a hypothesis test fails?

If $X\sim P$, given some other distribution $Q\gg P$ what is known about $\mathbb{P}(P(X)< Q(X))$, i.e. the probability the outcome was more likely to have come from $Q$? In particular are there ...
345 views

3k views

### What is the meaning of || (double vertical bar) in this KL divergence equation?

What is the meaning of the || in this equation? I haven't been able to find it from googling. It's from page 8 of https://arxiv.org/pdf/1606.05908.pdf Thanks!
20k views

### Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
148 views

### How to use Kullback-Leiber divergence for discriminating 2 keywords

I'm getting 500 tweets for each of the keywords K1 and K2 and I want to discriminate these 2 keywords with Kullback-Leiber Divergence formula. I do normal text mining preprocessing such as removing ...
351 views

### How do I compare collections of samples for differences?

I got data of the speech length distribution (number of words per speech) of a Speaker in a tv series per episode in my table. E.g.: ...
674 views

### KL Divergence, Bregman, and uniqueness

While reading the following paper on Bregman Divergence (link) Banerjee, Arindam, et al. "Clustering with Bregman divergences." Journal of machine learning research 6.Oct (2005): 1705-1749. In ...
207 views

### KL divergence in Sequential Monte Carlo

Suppose at step $t$ the particle approximation of SMC in $d$ dimensions is given by $\sum_{k=1}^N w_k\delta(\vec{x}-\vec{x}_k)$, and at the subsequent step, $t+1$ (after using Bayes' law to update ...
50 views

### Comparing approximating mixture distributions

Setup: Say I have some Bayesian predictive model I assume to be true for each observation $x_1$. Each $x_2$ is a latent/unseen/hidden random variable. The parameters are $\theta$. It's a mixture ...
1k views

### KL divergence between two bivariate Gaussian distribution

KL divergence between two multivariate Gaussians and univariate Gaussians have been discussed. I was wondering if there exists a simpler computation for the KL divergence between two bivariate ...
244 views

### variational bayes lower bound derivation for a Normal

I'm learning about variational inference and tried to follow the example for a Gaussian Mixture as described here. I could follow the bound for $\phi_k$ which is the difference of two Beta log ...
216 views

### KL divergence between discrete data and model (choosing hyperprior over Dirichlet concentration parameter $\alpha$)

I have some categorical data that follow an unknown true multinomial distribution $p$ and a model with known multinomial distribution $q$. I want to estimate the KL divergence between $p$ and $q$ ...
705 views

### textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
2k views

### Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...
1k views

### Comparing two PDFs using Kullback-Leibler divergence

I am trying to compare two PDFs using Kullback-Leibler divergence but I am getting a value which means they are almost identical. Am I missing something? Here is my code. ...
1k views

551 views

102 views

### Appropriate divergence measure for a distribution over ordinal values

I would like to measure the divergence (or, more appropriately, symmetric difference) between two distributions $P$ and $D$. In general, you could consider using a measure like Jensen-Shannon ...
420 views

### Approximating KL-Divergence for 2-D Random Variables with Scatter Plots

I have lots of experience computing KL divergences for straightforward discrete distributions where I have access to complete probability tables, etc. But I'm a little concerned about my current ...
345 views

### Monotonicity of special case of Kullback-Leibler divergence

I have two discrete distributions $\tau$ and $\rho$ with the same support $\Omega$. I'm considering a weighted mixture of these distributions described by the following function:  f(w) = (1-w) \cdot ...
I tried to implement a numerical estimate of the Kullback-Leibler Divergence for two samples. To debug the implementation draw the samples from two normal distributions $\mathcal N (0,1)$ and $\... 1answer 33 views ### Compare two distributions with varying focus on different regions I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ... 2answers 824 views ### What are the assumptions that we make when we compute KL Divergence between two distributions? Let us assume that we compute the KL Divergence between p and q. Is it necessary that both p and q belong to the exponential family of distributions. Moreover, is it necessary, that both p and q ... 0answers 217 views ### What tests can I use to compare these two probability distributions I am trying to compare two one-dimensional distributions. I am using Kullback-Leibler divergence function for this but it requires me to have both the distributions of equal length. I am not sure how ... 0answers 288 views ### How to compare posterior distributions for different observed data? KL-divergence? So I'm solving an inverse problem with the Bayesian approach$p(u | y) \propto p(y| u )p(u)$. Assuming I have two datasets$y_1$and$y_2$, what can be said about the difference in the posteriors$p(...
Dirichlet prior/posterior provides a probability density on distributions over a multinomial variable. It has the form : $P(P) \varpropto \prod_i{P_i^{\alpha_i-1}}$ I can also describe the ...