Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
1
vote
1answer
136 views

Choosing a Model Selection Criterion [closed]

I am trying to decide which among for model selection criteria to use for a Bayesian nonparametric model. The candidates are: The L-criterion, as defined by Laud & Ibrahim (1995); Bayes factors; ...
3
votes
0answers
39 views

What is the distribution G (with given mean and variance) that has the minimum KL distance from normal distribution F?

I was hoping someone knows whether the following statement is true or not. Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (...
9
votes
3answers
2k views

Estimate the Kullback Leibler (KL) divergence with monte carlo

I want to estimate the KL divergence between two continuous distributions f and g. However, I can't write down the density for either f or g. I can sample from both f and g via some method (for ...
2
votes
2answers
2k views

Kullback-Leibler distance for comparing two distribution from sample points

I have two data samples of a value and I want to compute some distance which would represent the difference in their distribution. I read about Kullback-Leibler distance which could be used for ...
5
votes
1answer
135 views

What is the probability that a hypothesis test fails?

If $X\sim P$, given some other distribution $Q\gg P$ what is known about $\mathbb{P}(P(X)< Q(X))$, i.e. the probability the outcome was more likely to have come from $Q$? In particular are there ...
2
votes
0answers
345 views

Deriving the gradient of the loss in SNE

The objective used in SNE is the KL divergence between the two distributions and is given as $$ E(Y) = \sum_i \sum_j p_{j|i}\log \frac{p_{j|i}}{q_{j|i}} $$ and the two distributions are as follows, $...
2
votes
1answer
3k views

How do I calculate KL-divergence between two multidimensional distributions? [closed]

Each distribution is represented with an array of arrays with PMF values. UPD 1: I have $P=(p_1, ... , p_n)$ where $P$ is a distribution of distributions and $p_i=(p_i^1, ..., p_i^m)$. My task is to ...
2
votes
0answers
309 views

KL divergence between function of two distributions

Suppose we have two random variables: $X \sim f(x)$ and $X' \sim f'(x)$ and we know how to compute $KL(f||f')$. We apply a one to one function $g$ on both random variables which gives us two new ...
1
vote
1answer
325 views

multidimensional KL loss

I read this question on Kullback Liebler Divergence Now i'm have a multidimensional distributions, like these: for example i try to predict if a person in image is a male: ...
3
votes
1answer
57 views

How does one quantify the difference between two distributions, especially if sample sizes differ?

I have plotted some experimental data of mine, and these data points fall into the following distributions: So, these are fairly non-trivial looking distributions. I would like to figure out methods ...
5
votes
2answers
1k views

What is information-theoric about the Kullback-Leibler divergence?

Statistical references often present the Kullback-Leibler and $f$-divergences as being information-theoric in nature. Some examples: The paper On Information and Sufficiency, by Kullback and Leibler (...
1
vote
1answer
1k views

Obtaining Shannon entropy from “KL-divergence to uniform distribution”

Suppose I have a probability distribution $P$, and suppose that $U$ is the uniform distribution on the same sample space. Then the KL divergence from P to U is $D_{KL}(U||P) = \sum_x u(x) \log\frac{u(...
2
votes
1answer
3k views

What is the meaning of || (double vertical bar) in this KL divergence equation?

What is the meaning of the || in this equation? I haven't been able to find it from googling. It's from page 8 of https://arxiv.org/pdf/1606.05908.pdf Thanks!
39
votes
1answer
20k views

Why do we use Kullback-Leibler divergence rather than cross entropy in the t-SNE objective function?

In my mind, KL divergence from sample distribution to true distribution is simply the difference between cross entropy and entropy. Why do we use cross entropy to be the cost function in many machine ...
3
votes
0answers
148 views

How to use Kullback-Leiber divergence for discriminating 2 keywords

I'm getting 500 tweets for each of the keywords K1 and K2 and I want to discriminate these 2 keywords with Kullback-Leiber Divergence formula. I do normal text mining preprocessing such as removing ...
4
votes
2answers
351 views

How do I compare collections of samples for differences?

I got data of the speech length distribution (number of words per speech) of a Speaker in a tv series per episode in my table. E.g.: ...
3
votes
1answer
674 views

KL Divergence, Bregman, and uniqueness

While reading the following paper on Bregman Divergence (link) Banerjee, Arindam, et al. "Clustering with Bregman divergences." Journal of machine learning research 6.Oct (2005): 1705-1749. In ...
2
votes
0answers
207 views

KL divergence in Sequential Monte Carlo

Suppose at step $t$ the particle approximation of SMC in $d$ dimensions is given by $\sum_{k=1}^N w_k\delta(\vec{x}-\vec{x}_k)$, and at the subsequent step, $t+1$ (after using Bayes' law to update ...
2
votes
0answers
50 views

Comparing approximating mixture distributions

Setup: Say I have some Bayesian predictive model I assume to be true for each observation $x_1$. Each $x_2$ is a latent/unseen/hidden random variable. The parameters are $\theta$. It's a mixture ...
3
votes
1answer
1k views

KL divergence between two bivariate Gaussian distribution

KL divergence between two multivariate Gaussians and univariate Gaussians have been discussed. I was wondering if there exists a simpler computation for the KL divergence between two bivariate ...
1
vote
1answer
244 views

variational bayes lower bound derivation for a Normal

I'm learning about variational inference and tried to follow the example for a Gaussian Mixture as described here. I could follow the bound for $\phi_k$ which is the difference of two Beta log ...
1
vote
0answers
216 views

KL divergence between discrete data and model (choosing hyperprior over Dirichlet concentration parameter $\alpha$)

I have some categorical data that follow an unknown true multinomial distribution $p$ and a model with known multinomial distribution $q$. I want to estimate the KL divergence between $p$ and $q$ ...
6
votes
2answers
705 views

textbook example of KL Divergence [duplicate]

I have read what KL Divergence is about: assess differences in probability distributions between two sets. I have also read, and digested, that it is emphatically not a true metric because of ...
23
votes
3answers
2k views

Kullback-Leibler divergence WITHOUT information theory

After much trawling of Cross Validated, I still don't feel like I'm any closer to understanding KL divergence outside of the realm of information theory. It's rather odd as somebody with a Math ...
3
votes
1answer
1k views

Comparing two PDFs using Kullback-Leibler divergence

I am trying to compare two PDFs using Kullback-Leibler divergence but I am getting a value which means they are almost identical. Am I missing something? Here is my code. ...
2
votes
1answer
1k views

Minimisation of Kullback-Leibler divergence on an arbitrary parameter

With the definition of the KL divergence between a model probability density function (pdf) and the data pdf $$D[p|q] = \bigg< log \frac{p(x)}{q(x)} \bigg>_p = \int_{-\infty}^{\infty} p(x) log \...
3
votes
1answer
461 views

KL divergence for a hierarchical prior structure e.g. Linear Regression

For a Linear Regression $\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \sigma^2\mathbb{I})$, suppose the prior set on $\beta_k$ is $\sim \mathcal{N}(0, l_k)$ ...
3
votes
1answer
153 views

Chow-Liu trees and Kullback Leibler divergence

I'm reading David Barber's book on Bayesian Reasoning and Machine Learning. At Section 9.5.4 he covers Chow-Liu trees, and I am having difficulties understanding the flow of the equations after he ...
2
votes
1answer
704 views

Comparing two word distributions

I want to compare two empirical distributions of words in two different texts to see if they are reasonably similar. So for each text I perform the usual steps like stopword removal and stemming, and ...
1
vote
1answer
75 views

Variational methods - explaining theory expression

Could anyone explain unrolling the following equation? I do not see how the difference is equal to Kullback-Leibler equation. $$ \mathcal{L} (\theta) - \mathcal{F}(q,x) = \log p(y|\theta) - \int{q(x) ...
7
votes
3answers
551 views

Choose probability distribution to maximize evaluation function (for CDC flu forecasting contest)

Suppose you have a discrete random variable $X$ with probability mass function $p(x) = P(X=x)$ on the support $0,\ldots,n$. What function $q(x)\ge 0$ such that $\sum_{x=0}^n q(x) = 1$ maximizes $$ E(\...
4
votes
2answers
4k views

How to use Kullback-leibler divergence if mean and standard deviation of of two Gaussian Distribution is provided?

With Apache Spark MLLib library I am trying to find Clusters within a dataset by using Gaussian Mixture Model (number cluster =3) . Now it returns 3 different values of mean and standard deviation. I ...
1
vote
0answers
69 views

How to evaluate the distance between PDFs without knowing the exact probability density function of one distribution

I have a set of random variables $X_1$, $X_2$, $X_3$, ... $X_n$. They are continuous and $0 \leq X_i \leq 1$. And Let's assume they are i.i.d from the same distribution. How do I evaluate the KL-...
1
vote
1answer
77 views

Is this the right interpretation of relative entropy for the Bayesian approach?

Suppose I know the true distribution, $P^*(x)$ and I have approximated the true distribution with $\tilde{P}(x|D)$, which is the predictive posterior density. Does the relative entropy of $\tilde{P}(x|...
3
votes
2answers
3k views

Kullback-Leibler divergence and probability distribution function in MATLAB

I want to compute the Kullback-Leibler divergence (KL) of two Gaussians, the first with mean of 1 and the second -1, where both have the same variance say, 1. In MATLAB, the distributions are: <...
3
votes
1answer
27 views

What is the factor equal to if the true and empirical distribution both are 0 for a configuration?

Suppose I want to calculate the relative entropy: $$D(q||p)= \sum q(x)\log \frac{q(x)}{p(x)}$$ If, for some $x$, I have $q(x)=p(x)=0$, does the corresponding factor in the sum becomes $0$?
0
votes
1answer
127 views

How to compute the KL-distance between 2 Bayesian networks?

Suppose I have a Bayesian network that can be factorized like this: P(A,B,C,D)=P(A)*P(B)*P(C|A,B)*P(D|C) Each of the variable is a binary and I've got all the tables of conditional probabilistic ...
1
vote
0answers
98 views

Kullback-Leibler divergence with sample data likelihood [duplicate]

I'm trying to get my head around the KL divergence in the context of the sample likelihood under two competing hypotheses, one optimal $H_0$ and one suboptimal $H_1$. Roughly speaking, I want to see ...
4
votes
2answers
557 views

KL divergence vs Absolute Difference between two distributions? [duplicate]

Why should I use KL divergence over just giving the abs difference from two PDFs?
1
vote
0answers
251 views

Derivation of autoencoders in backpropagation

I'm following the basics of autoencoders here: http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity Here are some of the important parts: But I don't understand the last part: Why is ...
6
votes
1answer
3k views

What is the intuition behind the Population Stability Index?

The "Population Stability Index" for two distributions $P$ and $Q$ is defined as the Symmetrised Kullback-Leibler divergence: $$ \mathrm{PSI}(P,Q) = D_{KL}(P||Q) + D_{KL}(Q||P) = \sum_i(P_i-Q_i)\log\...
2
votes
2answers
102 views

Appropriate divergence measure for a distribution over ordinal values

I would like to measure the divergence (or, more appropriately, symmetric difference) between two distributions $P$ and $D$. In general, you could consider using a measure like Jensen-Shannon ...
1
vote
1answer
420 views

Approximating KL-Divergence for 2-D Random Variables with Scatter Plots

I have lots of experience computing KL divergences for straightforward discrete distributions where I have access to complete probability tables, etc. But I'm a little concerned about my current ...
2
votes
1answer
345 views

Monotonicity of special case of Kullback-Leibler divergence

I have two discrete distributions $\tau$ and $\rho$ with the same support $\Omega$. I'm considering a weighted mixture of these distributions described by the following function: $$ f(w) = (1-w) \cdot ...
10
votes
2answers
7k views

Kullback-Leibler Divergence for two samples

I tried to implement a numerical estimate of the Kullback-Leibler Divergence for two samples. To debug the implementation draw the samples from two normal distributions $\mathcal N (0,1)$ and $\...
0
votes
1answer
33 views

Compare two distributions with varying focus on different regions

I have been trying to find if my problem matches has been discussed in prior research and if any technique exists to solve it. Here's the problem: Given two distributions (pdf) D1 and D2 over a ...
3
votes
2answers
824 views

What are the assumptions that we make when we compute KL Divergence between two distributions?

Let us assume that we compute the KL Divergence between p and q. Is it necessary that both p and q belong to the exponential family of distributions. Moreover, is it necessary, that both p and q ...
1
vote
0answers
217 views

What tests can I use to compare these two probability distributions

I am trying to compare two one-dimensional distributions. I am using Kullback-Leibler divergence function for this but it requires me to have both the distributions of equal length. I am not sure how ...
1
vote
0answers
288 views

How to compare posterior distributions for different observed data? KL-divergence?

So I'm solving an inverse problem with the Bayesian approach $p(u | y) \propto p(y| u )p(u)$. Assuming I have two datasets $y_1$ and $y_2$, what can be said about the difference in the posteriors $p(...
1
vote
0answers
162 views

Converting Dirichlet distribution to distribution on the log-linear parameters

Dirichlet prior/posterior provides a probability density on distributions over a multinomial variable. It has the form : $P(P) \varpropto \prod_i{P_i^{\alpha_i-1}}$ I can also describe the ...