Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

Filter by
Sorted by
Tagged with
4
votes
1answer
501 views

How to picture EM algorithm and KL-divergence geometrically?

In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition": In information geometry, the E step and the ...
14
votes
1answer
10k views

Why KL divergence is non-negative?

Why is KL divergence non-negative? From the perspective of information theory, I have such an intuitive understanding: Say there are two ensembles $A$ and $B$ which are composed of the same set of ...
2
votes
0answers
594 views

Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
6
votes
3answers
1k views

Why don't we use a symmetric cross-entropy loss?

Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be ...
2
votes
0answers
219 views

Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
0
votes
1answer
55 views

need help in understanding a research paper… specifically related to KL divergence

In this paper https://dl.acm.org/citation.cfm?id=2002654 specifically section 2.4. I understood that Review rating is modelled as random variable with guassian distribution with mean as wt * or. I ...
1
vote
0answers
340 views

Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
2
votes
1answer
270 views

How to justify a KL divergence when a distribution contains continuous and discrete components

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$ In these distributions, is there a way to justify a definition ...
2
votes
0answers
174 views

KL divergence and anomaly detection

stats newbie here. I have a dataset that is collected weekly. In order to make sure the data set gathered this week conform to past observations, I'm using KL divergence to compute how similar the ...
1
vote
0answers
61 views

Can we state that If KL-Divergence(P||Q) < H(P) then Q is “informative” of P and not otherwise?

From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence). I want to know ...
11
votes
1answer
994 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
0
votes
0answers
54 views

How to classify images with the same pattern, differing in color?

In order to increase your knowledge in the field of computer vision, I would like to create an application that will classify photos with the same pattern but with different colors. Is there any ...
1
vote
0answers
103 views

Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
5
votes
1answer
3k views

Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence, \begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation} And I ended up with a ...
2
votes
0answers
51 views

Inequality Kullback divergence

I have a problem with solving the following question. Let $\mathcal{P} = \{\mathbb{P}_\theta : \theta \in \Theta\}$ be a statistical family of discrete distributions with state space $\mathcal{X}$ ...
1
vote
0answers
332 views

Bhattacharyya distance and KL divergence show contradicting behavior

Bhattacharyya distance between two distributions $p$ and $q$ is defined as $D_B(p,q)=-\log(\int\sqrt{p(x)q(x)})dx$, The KL-divergence is defined as $D_{kl}(p||q)=\int p(x)\log(\frac{p(x)}{q(x)})dx$. ...
4
votes
0answers
869 views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
5
votes
1answer
2k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
9
votes
2answers
3k views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
1
vote
0answers
280 views

Is there an analytical solution for the Kullback-Leibler-Divergence with two univariate Lognormal distributions? [duplicate]

The KL is given by: $D_{\mathrm{KL}}(P\|Q) = \int_X p \, \log \frac{p}{q} \, d\mu.$ The PDF of a Lognormal distribution is given by: $P = \frac 1 x \cdot \frac 1 {\sigma\sqrt{2\pi\,}} \exp\left( -\...
2
votes
1answer
476 views

Kullback–Leibler divergence

I am watching this great lecture by Nando De Freitas. He establishes the KL divergence by using maximum liklihood estimation. However, there is one step I don't really understand. I do ...
3
votes
1answer
231 views

Does it make sense to directly subtract the entropy values of two distributions instead of using measures such as KL divergence or cross entropy?

My supervisor has written some (relatively draft-like) R code that implements an idea from Optimal Experiment Design. However, during instead of using KL divergence $D_{KL}(P(M | Y = y) || P(M))$ as ...
3
votes
1answer
472 views

KL divergence term in VAEs

I was reading the textvae paper (A Hybrid Convolutional Variational Autoencoder for Text Generation). There are some things in the paper that are counter-intuitive to me. First, "In most cases the ...
2
votes
2answers
2k views

Kullback-Leibler divergence lower bound

Are there any (nontrivial) lower bounds on the KL-divergence between two densities? Informally, I am trying to study problems where $f$ is some target density, and I want to show that if $g$ is chosen ...
1
vote
0answers
77 views

Scoring “predictions” using Kullback-Leibler

I'm interested in the following problem: I have a biased coin, which you measure, and make an estimate $p_{est}$ of the odds of getting a heads. I then flip the coin a random (say, Poisson(10)) ...
1
vote
0answers
35 views

Understanding Natural gradient learning Independent component analysis

I am quite a novice to statistics and am currently fighting my way through the "Neural Networks and Machine Learning" Book by Haykin. (p.516-518) In the discussion about independent component analysis ...
1
vote
1answer
2k views

Jensen-Shannon Divergence for multiple probability distributions?

What is the correct mathematical expression for computing Jensen-Shannon divergence between multiple probability distributions? I found the following expression on Wikipedia, but I did not find any ...
5
votes
2answers
493 views

If KL-divergence's asymmetrical should I minimize KL(P||Q) or KL(Q||P)?

Suppose I am adjusting distribution $Q$ to make get a best fit for distribution $P$, should I minimize $KL(P||Q)$ or $KL(Q||P)$? What is the difference? Related question: Intuition on the Kullback-...
0
votes
0answers
61 views

Lower bound for difference of probabilities of a same event under two different distributions

Say $P$ and $Q$ are two probabilities distributions on $[n]$. I can upper bound the difference of the probability of an event $A$ under $P$ and $Q$ by the total variation distance between $P$ and $Q$. ...
5
votes
1answer
412 views

Where does the Kullback-Leibler come from? [duplicate]

Let $p(x)$ be some "true" distribution which we want to model using a simpler distribution $q(x)$. Why is the KL divergence $$KL(q||p)=\int q(x)\log{\frac{q(x)}{p(x)}}$$ a good way to represents the ...
0
votes
1answer
66 views

Why W is transposed in the final form of multiplicative updates of NMF - $H^*=H * \frac{W^T\frac{V}{WH}}{W^T 1}$

For NMF using Kullback-Liebler divergence \begin{equation*} d_{\mathrm{KL}} (\mathbf{V}\ \vert\vert \mathbf{WH})\stackrel{\hbox{cst.}}{\hbox{=}} \sum_{ij}-V_{ij}\log\sum_{k}{(WH)_{ij}} + \sum_{ij}\...
1
vote
0answers
36 views

Information gain obtained by updating knowledge on “possible values” of a R.V

Let's assume that one does not know all possible values of a random variable $X$. He/she only knows that $X$ can be $1$ with probability $p(1)$ (and so $P(X \ne 1) = 1 - p(1))$, but does not know what ...
0
votes
0answers
76 views

Derive a constant in Kullback-Liebler divergence proof

From Kullback-Liebler divergence of matrix factorization; \begin{equation*} \mathrm{X}\approx\mathbf{WH} \tag{1} \end{equation*} How equation $(2)$ is derived to constant equality in equation $(3)$? ...
23
votes
2answers
5k views

What is the advantages of Wasserstein metric compared to Kullback-Leibler divergence?

What is the practical difference between Wasserstein metric and Kullback-Leibler divergence? Wasserstein metric is also referred to as Earth mover's distance. From Wikipedia: Wasserstein (or ...
2
votes
0answers
83 views

Interpreting base measure in exponential family as an improper prior (because entropy)

Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base ...
1
vote
0answers
142 views

Covariance matrix of two equiprobable classes

Today in class the professor gave us the following question: Two equiprobable classes with mean vectors μ1 = [4, 2]^T and ...
5
votes
0answers
1k views

What is the KL divergence of distribution from Dirac delta?

The Kullback–Leibler (KL) divergence of two continuous distributions $P(x)$ and $Q(x)$ is defined as $$D_{KL}(P \mid\mid Q) = \int_{X} P(x) \log{\left[\frac{P(x)}{Q(x)}\right]} dx$$ How can one ...
1
vote
1answer
825 views

kullback leibler divergence between two nested logistic regression models

I have two logistic nested models $\log\dfrac{p_i}{1-p_i}=\beta_{0}+\beta_1 x_i$ and $\log\dfrac{p_i}{1-p_i}=\beta_{0}$ How can I construct the kullback leibler divergence between two nested ...
2
votes
1answer
802 views

KL divergence of multivariate lognormal distributions

I've been trying to get the KL divergence for two lognormal distributions. I know what it is for the univariate case, $$ D(f_i\|f_j)= \frac1{2\sigma_j^2}\left[(\mu_i-\mu_j)^2+\sigma_i^2-\sigma_j^2\...
4
votes
1answer
1k views

KL divergence bounds square of L1 norm

In Cover & Thomas, Elements of Information Theory, at the section on Conditional Limit Theorem (11.6), it is proved that the KL divergence bounds the $\cal{L}_1$-norm from above, $\frac{1}{2\ln2}\...
1
vote
1answer
123 views

Choosing a Model Selection Criterion [closed]

I am trying to decide which among for model selection criteria to use for a Bayesian nonparametric model. The candidates are: The L-criterion, as defined by Laud & Ibrahim (1995); Bayes factors; ...
3
votes
0answers
39 views

What is the distribution G (with given mean and variance) that has the minimum KL distance from normal distribution F?

I was hoping someone knows whether the following statement is true or not. Suppose $F$ is a given normal distribution and $G$ is a distribution that has a given mean $\mu$ and variance $\sigma^2$ (...
9
votes
3answers
2k views

Estimate the Kullback Leibler (KL) divergence with monte carlo

I want to estimate the KL divergence between two continuous distributions f and g. However, I can't write down the density for either f or g. I can sample from both f and g via some method (for ...
2
votes
2answers
2k views

Kullback-Leibler distance for comparing two distribution from sample points

I have two data samples of a value and I want to compute some distance which would represent the difference in their distribution. I read about Kullback-Leibler distance which could be used for ...
5
votes
1answer
135 views

What is the probability that a hypothesis test fails?

If $X\sim P$, given some other distribution $Q\gg P$ what is known about $\mathbb{P}(P(X)< Q(X))$, i.e. the probability the outcome was more likely to have come from $Q$? In particular are there ...
2
votes
0answers
331 views

Deriving the gradient of the loss in SNE

The objective used in SNE is the KL divergence between the two distributions and is given as $$ E(Y) = \sum_i \sum_j p_{j|i}\log \frac{p_{j|i}}{q_{j|i}} $$ and the two distributions are as follows, $...
2
votes
1answer
3k views

How do I calculate KL-divergence between two multidimensional distributions? [closed]

Each distribution is represented with an array of arrays with PMF values. UPD 1: I have $P=(p_1, ... , p_n)$ where $P$ is a distribution of distributions and $p_i=(p_i^1, ..., p_i^m)$. My task is to ...
2
votes
0answers
304 views

KL divergence between function of two distributions

Suppose we have two random variables: $X \sim f(x)$ and $X' \sim f'(x)$ and we know how to compute $KL(f||f')$. We apply a one to one function $g$ on both random variables which gives us two new ...
1
vote
1answer
312 views

multidimensional KL loss

I read this question on Kullback Liebler Divergence Now i'm have a multidimensional distributions, like these: for example i try to predict if a person in image is a male: ...
3
votes
1answer
55 views

How does one quantify the difference between two distributions, especially if sample sizes differ?

I have plotted some experimental data of mine, and these data points fall into the following distributions: So, these are fairly non-trivial looking distributions. I would like to figure out methods ...