# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

122 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1k views

### What is the KL divergence of distribution from Dirac delta?

The Kullback–Leibler (KL) divergence of two continuous distributions $P(x)$ and $Q(x)$ is defined as $$D_{KL}(P \mid\mid Q) = \int_{X} P(x) \log{\left[\frac{P(x)}{Q(x)}\right]} dx$$ How can one ...
217 views

210 views

### Quantify the information lost given by the Kullback-Leibler divergence measure

Consider there are $N$ individuals and these measure a quantity $X\in \mathbb{R}^{N\times M}$ where $M$ is the number of measurements and let $P(X)$ denote a probability distribution over $X$. The ...
219 views

### Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]],$$ where $L$ is some loss function and $\hat x$ is the ...
129 views

### Does this bounded, continuous probability distribution have a name?

Does this bounded, continuous probability distribution over $x$ have a name? $P(x|y) \propto \big(\frac{y}{x}\big)^x\big(\frac{1-y}{1-x}\big)^{(1-x)}$ for $x, y \in (0,1)$. This comes about by ...
39 views

### What is the distribution G (with given mean and variance) that has the minimum KL distance from normal distribution F?

I was hoping someone knows whether the following statement is true or not. Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (...
147 views

### How to use Kullback-Leiber divergence for discriminating 2 keywords

I'm getting 500 tweets for each of the keywords K1 and K2 and I want to discriminate these 2 keywords with Kullback-Leiber Divergence formula. I do normal text mining preprocessing such as removing ...
80 views

### Properties of Average Multinomial Likelihood

I am trying to understand the Kullback-Leibler Information: I read in http://arxiv.org/pdf/1404.2000v1.pdf the following: Ideally, we want the probability to be invariant to the number of ...
89 views

### Name of an $f$-divergence

The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$. The relative ...
82 views

### Properties of the KL topology [reference request]

I'm trying to understand better what are the implications of a sequence of random variables $X_n$ converging toward some limit $X$ in the KL topology, ie the probability density functions are such ...
400 views

### How to use KL-divergence in naive bayes classifier to weight features?

I have a dataset consisting of 4 classes. I have implemented the Gaussian Naive Classifier (in Matlab). In the training phase I calculate the mean and variance for each feature and each class as well ...
2k views

### Under what conditions will Kullback-Leibler divergence/mutual information be infinity?

For two perfectly correlated Gaussian variables, the mutual information between them, and thus the KL divergence between the product of the marginal distributions and the joint distribution, is ...
187 views

### Formal statistical test for comparing likelihood distributions obtained via MCMC

I am trying to formally compare the distribution of the likelihood values generated using two different models with marginal posterior values of the parameters obtained using MCMC in order to assess ...