Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

An asymmetric measure of distance (or dissimilarity) between probability distributions. It might be interpreted as the expected value of the log likelihood ratio under the alternative hypothesis.

2
votes
0answers
13 views

KL divergence between which distributions could be infinity

I know that KL divergence measures difference between two probability distributions. My doubt is for which of the distributions it could become Infinity, putting it in another way, ...
0
votes
0answers
29 views

Kullback–Leibler divergence in multidimensional multinomial distribution [on hold]

Updated: I have two probability distributions P and Q, I want to calculate the KL divergence of them. P = [2/7, 5/7, 3/7, 2/7, 2/7], Q = [2/11, 9/11, 3/11, 4/11, 4/11]. Each vector is combined by ...
3
votes
1answer
67 views

What is the difference Cross-entropy and KL divergence?

Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference? $$ H(P,Q) = -\sum_x P(x)\log Q(x) $$ $$ KL(P | Q) = \sum_{x} P(...
0
votes
1answer
70 views

Rate of converge of KL-divergence to posterior

Suppose you have samples from some distribution P. You have a prior distribution Q, which represents your estimate of P, and assume for now that it's parameterized the same way as P. Upon observing ...
0
votes
1answer
39 views

Difference between Empirical distribution and Bernoulli distribution

I've been studying binary cross entropy error for binary classification weight optimization. From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with ...
1
vote
1answer
54 views

Why Kullback-Leibler in Stochastic Neighbor Embedding

Stochastic Neighbor Embedding (and t-SNE) relies on Kullback-Leibler divergence between the point distributions in the original and the low-dimensional space. Why? Why not any other dissimilarity ...
1
vote
0answers
32 views

Choosing an appropriate statistical distance which punishes entropy

Problem Description: I have with me experimental statistics of a system and I wish to fit a theoretical model so that the computed statistics on the model fit the experimental ones. I am using an RBM ...
0
votes
0answers
23 views

Similarity of two sets of points

PROBLEM I have one set of 10 points, $X = \{(x_1,y_1),\,\dots,\,(x_{10},y_{10})\}$ and two sets of 3 points each, $A = \{ (a_1,b_1),\, (a_2,b_2),\, (a_3,b_3) \}$ and $C = \{(c_1,d_1),\,(c_2,d_2),\,(...
11
votes
3answers
941 views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
4
votes
1answer
42 views

Comparing distribution A to B and C

I have three discrete probability distributions, A, B and C. They are all measuring P(X) under different circumstances. I suspect that A is more similar to B than it is to C. I know that I can compare ...
1
vote
0answers
67 views

Kullback–Leibler divergence when one measure is a sum of diracs

In the book "Deep Learning" of Goodfellow, Bengio and Courville, section 5.5 of maximum likelihood estimation they explain a relation between the maximization of likelihood and minimization of the K-L ...
2
votes
0answers
32 views

What is the intuition regarding the different viewpoints of the Kullback-Leibler Divergence and the Kolmogorov-Smirnov Statistic? [duplicate]

In trying to understand the Kullback-Leibler Divergence I conceive it as a metric that if minimized would make the Approximation PDF Q as close as possible to the True PDF P either in the ...
0
votes
0answers
97 views

Calculating Jensen-Shannon and Kullback-Leibler for density splines

I need to find a way to calculate the JS distance (and, by necessity, the KL divergence) between continuous, non-normal (generating distribution unknown) distributions. I have many classes, and ...
3
votes
1answer
121 views

Does this bounded, continuous probability distribution have a name?

Does this bounded, continuous probability distribution over $x$ have a name? $P(x|y) \propto \big(\frac{y}{x}\big)^x\big(\frac{1-y}{1-x}\big)^{(1-x)}$ for $x, y \in (0,1)$. This comes about by ...
2
votes
0answers
59 views

If the AIC is an estimate of Kullback-Leibler Divergence, then why can AIC be negative when KL divergence is always positive?

I have read many times that the AIC serves as an estimate of the KL divergence, and I know that AIC can be a negative value (and have seen that myself). Yet, the KL divergence must always be positive. ...
2
votes
2answers
99 views

KL divergence invariant to affine transformation?

I read in this tutorial on page 20 that $KL$ divergence is invariant to affine transformation, but I think it is incorrect. Say we have two 1D normal distributions $P_{1}(x) = \mathcal N(\mu_{1}, \...
4
votes
1answer
73 views

KL Divergence between parallel lines

I am trying to understand the example described in the WGAN paper about learning parallel lines with various divergences. More specifically the setup is as follows: Let $Z \sim [0, 1]$ the uniform ...
2
votes
3answers
65 views

Simulating KL Divergence between Cauchy RV and the MLE estimate of the RV - Multimodality seems wrong

I'm working on a (what I think is a fairly simple, straightforward) explanation of how it's really hard to approximate distributions with fat tails accurately in the tails. I started looking at an ...
5
votes
1answer
282 views

What constitutes a large KL divergence?

I have 2 gamma distributions $X_1 \sim Ga(13,1) \\ X_2 \sim Ga(3,1)$ Where we are defining our gamma distribution probability density function for a random variable $X \sim Ga(a,b)$ to be $f(X) = \...
0
votes
0answers
14 views

Generating distributions specifically distanced from the ground truth - (Betting strategy simulation)

Thanks in advance for your patience and helpful answers. I am new to cross validated so hopefully I can word this correctly. I am developing a betting strategy and I'd like to use a rigorous testing ...
3
votes
1answer
53 views

Connection between log predictive density and Kullback-Leibler information measure

I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information. The log ...
1
vote
1answer
128 views

Calculate the Kullback-Leibler Divergence for these 2 Gamma distributions

I have 2 models $P \sim Ga(115,1329.914) \\ Q \sim Ga(140,650.6775)$ and I'm looking to calculate the K-L divergence of these 2. $D_{KL}(P||Q) = \int_\infty ^\infty p(x) log \frac{p(x)}{q(x)}\,dx$...
4
votes
1answer
108 views

How to picture EM algorithm and KL-divergence geometrically?

In reading up on the Expectation-Mmaximization algorithm on Wikipedia, I read this short and intriguing line, under the subheading "Geometric Intuition": In information geometry, the E step and the ...
3
votes
1answer
996 views

Why KL divergence is non-negative?

Why is KL divergence non-negative? From the perspective of information theory, I have such an intuitive understanding: Say there are two ensembles $A$ and $B$ which are composed of the same set of ...
0
votes
0answers
111 views

Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
1
vote
2answers
242 views

Why don't we use a symmetric cross-entropy loss?

Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be ...
1
vote
0answers
65 views

Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
0
votes
1answer
47 views

need help in understanding a research paper… specifically related to KL divergence

In this paper https://dl.acm.org/citation.cfm?id=2002654 specifically section 2.4. I understood that Review rating is modelled as random variable with guassian distribution with mean as wt * or. I ...
0
votes
0answers
97 views

Comparing ELBO of a VAE for different samples

I am lacking of an interpretation of the evidence lower bound (ELBO), when comparing two different samples $x_1, x_2 \sim X$. Writing the marginal log-likelihood as the sum of lower variational bound ...
1
vote
1answer
90 views

How to justify a KL divergence when a distribution contains continuous and discrete components

Like in contamination models, some distributions have discrete component. e.g. $p(x) := (1 - \epsilon) q(x) + \epsilon \delta_{x_0}(x)$ In these distributions, is there a way to justify a definition ...
0
votes
0answers
15 views

Algorithm for highly correlated columns in fixed dictionary for Kullback-Leibler divergence problem?

According to this paper, l1-regularized linear least-squares problem with fixed dictionary is efficient. But if the columns of dictionaries are in highly correlated, LARS-Lasso algorithm provides ...
0
votes
0answers
49 views

Estimating the Kullback-Leibler divergence between exact distribution and its M-projection

My question is quite general. But, I prefer to illustrate it with the actual simple problem I am facing. Suppose that I have a naive Bayesian network (a star-like model). I want to marginalize out ...
1
vote
0answers
92 views

KL divergence and anomaly detection

stats newbie here. I have a dataset that is collected weekly. In order to make sure the data set gathered this week conform to past observations, I'm using KL divergence to compute how similar the ...
0
votes
0answers
17 views

Comparing two distribution

I have a distribution $p$ and estimate of that distribution $q$. I want to say that my estimate $q$ is good. I can see from the plots that it traces my real distribution. But how can I justify this ...
0
votes
0answers
32 views

Calculating if multivariable samples are drawn from similar distributions. (Kullback–Leibler divergence?)?

I have a list of related examples from a given set. Each sample is characterized by a number of different (real-valued) properties. I also have a list of other samples (all related to each other), ...
1
vote
0answers
42 views

Can we state that If KL-Divergence(P||Q) < H(P) then Q is “informative” of P and not otherwise?

From what I've read the KL-Divergence between $P||Q$ is the extra amount of "bits" you need to describe $P$ if you are encoding it with $Q$.(Analysis of Kullback-Leibler divergence). I want to know ...
11
votes
1answer
587 views

Interpretation of Radon-Nikodym derivative between probability measures?

I have seen at some points the use of the Radon-Nikodym derivative of one probability measure with respect to another, most notably in the Kullback-Leibler divergence, where it is the derivative of ...
0
votes
0answers
32 views

How to classify images with the same pattern, differing in color?

In order to increase your knowledge in the field of computer vision, I would like to create an application that will classify photos with the same pattern but with different colors. Is there any ...
1
vote
0answers
42 views

Questions about Mean-field variational inference

I am very new to this variational inference concept. I couldn't find any clear sources. I have two questions related to each other. Let's consider a very simple probabilistic model with a 2-D latent ...
0
votes
0answers
52 views

Ratio of Entropy and Kullback-Leibler divergence

Does the following measure or its inverse have a name? \begin{equation} \frac{H\big[p_{1}(x)\big]}{D_{KL}\big(p_{1}(x) || p_{2}(x)\big)} \end{equation}
5
votes
1answer
1k views

Can KL-Divergence ever be greater than 1?

I've been working on building some test statistics based on the KL-Divergence, \begin{equation} D_{KL}(p \| q) = \sum_i p(i) \log\left(\frac{p(i)}{q(i)}\right), \end{equation} And I ended up with a ...
2
votes
0answers
49 views

Inequality Kullback divergence

I have a problem with solving the following question. Let $\mathcal{P} = \{\mathbb{P}_\theta : \theta \in \Theta\}$ be a statistical family of discrete distributions with state space $\mathcal{X}$ ...
0
votes
0answers
28 views

How to calculate AIC for function $f(x)=\text{sign}(\sin(tx)), \,t>0$

I'm using a binary classifier function: $$f(x)=\text{sign}(\sin(tx)), \,t>0$$ which I want to fit to some data set $(x_1,y_1), ...(x_n, y_n)$, where $x_i\in\mathbb{R}, y_i\in\{-1, 1\},\;\forall ...
0
votes
0answers
124 views

Bhattacharyya distance and KL divergence show contradicting behavior

Bhattacharyya distance between two distributions $p$ and $q$ is defined as $D_B(p,q)=-\log(\int\sqrt{p(x)q(x)})dx$, The KL-divergence is defined as $D_{kl}(p||q)=\int p(x)\log(\frac{p(x)}{q(x)})dx$. ...
0
votes
0answers
12 views

Are the information criteria applicable only for some specific estimators?

I'm studying about information criteria (IC) and how they are used to estimate the Kullback-Leibler (KL) -information (or divergence) of an estimator distribution $F$ and a true data generating ...
3
votes
0answers
342 views

Maximum Likelihood & Bayesian inference minimizing Kullback-Leibler divergence?

I have heard/read that Bayesian and Maximum Likelihood inference can be justified as asymptotically minmizing the KL divergence between the pdf $p(x)$ actually describing the data and the ...
0
votes
0answers
430 views

Deriving the KL divergence loss for VAEs

In a VAE, the encoder learns to output two vectors: $$\mathbf{\mu} \in\ \mathbb{R}^{z}$$ $$\mathbf{\sigma} \in\ \mathbb{R}^{z}$$ which are the mean and variances for the latent vector $\mathbf{z}$, ...
5
votes
1answer
511 views

KL Loss with a unit Gaussian

I've been implementing a VAE and I've noticed two different implementations online of the simplified univariate gaussian KL divergence. The original divergence as per here is $$ KL_{loss}=\log(\frac{\...
1
vote
0answers
53 views

Is there an analytical solution for the Kullback-Leibler-Divergence with two univariate Lognormal distributions? [duplicate]

The KL is given by: $D_{\mathrm{KL}}(P\|Q) = \int_X p \, \log \frac{p}{q} \, d\mu.$ The PDF of a Lognormal distribution is given by: $P = \frac 1 x \cdot \frac 1 {\sigma\sqrt{2\pi\,}} \exp\left( -\...
1
vote
1answer
170 views

Kullback–Leibler divergence

I am watching this great lecture by Nando De Freitas. He establishes the KL divergence by using maximum liklihood estimation. However, there is one step I don't really understand. I do ...