Questions tagged [kullback-leibler]

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

302 questions
Filter by
Sorted by
Tagged with
501 views

How to picture EM algorithm and KL-divergence geometrically?

In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition": In information geometry, the E step and the ...
10k views

Why KL divergence is non-negative?

Why is KL divergence non-negative? From the perspective of information theory, I have such an intuitive understanding: Say there are two ensembles $A$ and $B$ which are composed of the same set of ...
594 views

Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
1k views

Why don't we use a symmetric cross-entropy loss?

Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be ...
219 views

Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
55 views

need help in understanding a research paper… specifically related to KL divergence

In this paper https://dl.acm.org/citation.cfm?id=2002654 specifically section 2.4. I understood that Review rating is modelled as random variable with guassian distribution with mean as wt * or. I ...
340 views

Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
270 views

How to justify a KL divergence when a distribution contains continuous and discrete components

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$ In these distributions, is there a way to justify a definition ...
174 views

KL divergence and anomaly detection

stats newbie here. I have a dataset that is collected weekly. In order to make sure the data set gathered this week conform to past observations, I'm using KL divergence to compute how similar the ...
61 views

Can we state that If KL-Divergence(P||Q) < H(P) then Q is “informative” of P and not otherwise?

From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence). I want to know ...
994 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
54 views

How to classify images with the same pattern, differing in color?

In order to increase your knowledge in the field of computer vision, I would like to create an application that will classify photos with the same pattern but with different colors. Is there any ...
103 views

Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
3k views

Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence, \begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation} And I ended up with a ...
51 views

Inequality Kullback divergence

I have a problem with solving the following question. Let $\mathcal{P} = \{\mathbb{P}_\theta : \theta \in \Theta\}$ be a statistical family of discrete distributions with state space $\mathcal{X}$ ...
332 views

Bhattacharyya distance and KL divergence show contradicting behavior

Bhattacharyya distance between two distributions $p$ and $q$ is defined as $D_B(p,q)=-\log(\int\sqrt{p(x)q(x)})dx$, The KL-divergence is defined as $D_{kl}(p||q)=\int p(x)\log(\frac{p(x)}{q(x)})dx$. ...
869 views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
2k views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
3k views

1k views

3k views

How do I calculate KL-divergence between two multidimensional distributions? [closed]

Each distribution is represented with an array of arrays with PMF values. UPD 1: I have $P=(p_1, ... , p_n)$ where $P$ is a distribution of distributions and $p_i=(p_i^1, ..., p_i^m)$. My task is to ...
304 views

KL divergence between function of two distributions

Suppose we have two random variables: $X \sim f(x)$ and $X' \sim f'(x)$ and we know how to compute $KL(f||f')$. We apply a one to one function $g$ on both random variables which gives us two new ...