# Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

370 questions
Filter by
Sorted by
Tagged with
23 views

### Does minimizing KL imply boundedness of density ratios

Suppose $q^*(\theta) = \underset{q \in \mathcal{Q}}{argmin} \,\, KL[q(\theta)||p(\theta)] = \min_{q\in\mathcal{Q}} \int_{\theta\in\Theta}q(\theta)\ln\Big( \frac{q(\theta)}{p(\theta)} \Big)d\theta$ ...
25 views

### Meaning of entropy for a multivariate data set

In an earlier question, I asked, “Any meaning to the concept of ‘Self Mutual Information?” and got a great answer - thanks. Now, this begs another question/set of questions: What does the concept of ...
146 views

56 views

### Clarifying notation for the Kullback-Leibler divergence in terms of expectations

We know that for discrete variables \begin{equation} D(p(x),q(x))=\mathbb{E}_{p}\left(\log\frac{p(x)}{q(x)}\right) \end{equation} where $p(x)$ and $q(x)$ are probability mass functions. Can this be ...
49 views

### Distance measure for two probability distribution of unequal sample size

Context: I have 100 stores and these stores are divided into 10 business markets. I want to select 3 markets where each market is a good representation of the 100 stores i.e. the population. There ...
28 views

### Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
163 views

### KL divergence between two Asymmetric Laplace distributions?

Consider two asymmetric Laplace distribution $L_1(\mu_1,\sigma_1,\tau_1)$ and $L_2(\mu_2,\sigma_2,\tau_2)$ where \begin{equation} L(x;\mu,\sigma,\tau) =\frac{\tau(1-\tau)}{\sigma} \begin{cases} ...
64 views

### Which KL Divergence is larger D(P|Q) or D(Q|P)?

From the perspective of information theory, I understand how D(P|Q) is non-negative and why the KL divergence is asymmetric, i.e. $D(P|Q) \neq D(Q|P)$, given two gaussian univariate gaussian ...
37 views

### Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution

I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says One way to interpret maximum likelihood estimation is to view it as minimizing the dissimilarity between the empirical ...
11 views

### Computing KL Divergence for distributions over sets

I have a distribution over a set of (hundreds of) discrete terms, and I'd like to describe the difference between I see a couple of options, and none seems really attractive: Take the KL divergence ...
597 views

### Gradients of KL divergence and ELBO for variational inference

When doing variational inference, due to intractability we typically maximize the evidence lower bound (ELBO) instead of minimizing Kullback-Leibler divergence (KLD) between our approximate and exact ...
25 views

427 views

### Meaning of expectation with respect to a function?

This might be a trivial question, but I've come across a paper where some expectation is said to be taken with respect to some pdf. See example: How am I to interpret this, and is there some ...
92 views

### Variational inference with deterministic dependencies between variables

Suppose I have a probabilistic graphical model shown in the picture, in which all variables are binary, $c_1$ and $c_2$ are observed, and I want to use mean-field variational inference to estimate ...
98 views

177 views

### How to use KL divergence to compare two distributions?

I am trying to model the probability distribution of a multi-dimensional dataset where all the values are discrete. Suppose the training data (represented by T) is of the shape (m, n) where n is the ...
So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution: $$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & ... 0answers 30 views ### How to prove KL(q(z)|p(z)) = E_q(z) [ log f(z) ] where f is the CDF of p? As title. It was used in https://arxiv.org/abs/1905.10549 without proving. 0answers 9 views ### Markov type bound based on KL divergence? Given two discrete distribution p, q on some universe U, if I know they have a bounded KL divergence, say some number c, can I say anything about how much each point in the universe differs in ... 1answer 1k views ### Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate] I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-... 0answers 50 views ### A divergence that can be extended to logistic functions? If I have data \{(x_i, y_i)\}_{i=1}^n where the dependent variable is binary (y_i = 0,1) I can model it using a logistic function:$$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}} ...
Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...