# Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

52 questions
Filter by
Sorted by
Tagged with
1 vote
20 views

6 views

### What does the corresponding f(t) mean with respect to f-divergences?

In terms of f-divergences, what does the corresponding $f(t)$ refer to, what does it mean and why is it important? For example, with the KL-divergence the corresponding $f(t)$ is ${t \ log \ t}$. I ...
28 views

### How to find the 'distance' between two populations?

I am somewhat new to these concepts, so please bear with me. I have two datasets: Data set A is collected by monitoring the network data of a device when it is ...
12 views

### using divergence for comparing more than two data sets(100s) with varying number of data points

I have 100 companies and for each company, I have measured a feature of employees. The number of employees measured is not the same in all companies (the size of the companies can vary a lot). The ...
18 views

### VAE divergence is positive in minimization of variational inference?

I have been going through the minimization of Variational inference and have a good understanding of all the steps taken: However, there is a part that relies on KL >= 0: I have derived the ...
11 views

### Different notion of stationarity for distribution shift

Consider a stochastic process $(X_t)_{t\in T}$. Usually this will be a time series, so for simplicity consider $T=\mathbb{N}$ (or $T=\mathbb{Z}$). For many applications this process will be ...
10 views

### Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
94 views

### What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age). I would ...
1 vote
80 views

### Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence: H(P, Q) = H(P) + D_KL(P||Q) Minimizing Cross-entropy means minimizing ...
25 views

### Measure for evaluating a density estimation procedure

Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure? I am currently evaluating the procedure using three ...
1 vote
60 views

### $\sqrt{2 KL(f || g)}$ interpretation?

I have seen in some papers that instead of using the Kullback-Leibler divergence $KL(f || g)$ between two probability density functions, $f$ and $g$, they use $$\sqrt{2 KL(f || g)}.$$ Is there any ...
58 views

### Estimating the $\chi^2$-divergence with Monte Carlo: which distribution to sample from?

Notation: let the $\chi^2$-divergence between $p, q$ be defined as $$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x -1 = \int \frac{p(x)}{q(x)} p(x)\mathrm{d}x - 1.$$ ...
28 views

### How do I interpolate a field that is divergence-free and curl-free at the same time?

A magnetic field is divergence free. At the points where there is no current, and no changing electric field, it is also curl free. There exist divergence-free and curl-free RBF kernels, and I could ...
1 vote
34 views

### Measuring the divergence in centrality statistics for similar networks?

I want to measure the similarity/divergence between the centrality of nodes in two publicly available word association networks. In my analysis, we have a long list of nodes - 12,000 or so - and then ...
14 views

### Grouping most strongly contributing items between different samples

I am performing an analysis by using Python's collections.Counter to obtain a scale of frequency x rank. I obtained the same statistics for multiple datasets and ...
80 views

### Formal arguments for why an asymmetric f-divergence might be favourable to a symmetric one in analyzing importance sampling

I am reading Importance Sampling and Necessary Sample Size: an Information Theory Approach. Below is a quote from paragraph 3, section 3 of the article. While [total variation distance] and [...
1 vote
80 views

### Why is reverse KL more suited for data generation

Here goes a first question! In a paper I'm reading in the context of GAN's (WGAN in particular) I came across the following quote when the authors discuss KL divergence: while maximum likelihood ...
33 views

### Comparing two noisy probability measurements

I have two sampled probability values (sequence P = [p0, p1..pn] and sequence Q = [q0, q1,...qn]. Both of them are evaluated on time t0, t1...tn (equidistant). For simplicity, P is probability that ...
553 views

### What is meant by divergence in statistics?

I have learned about the Intuition on the Kullback-Leibler (KL) Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The two most important ...
1 vote
29 views

### estimate the divergence between two distributions by comparing the sufficient statistics

I'm interested in comparing two distributions $p(x,y)$ and $q(x,y)=pr(x)q(y|x)$. I want to estimate the KL-divergence between $p(x,y)$ and $q(x,y)$. It could be other divergence too, and I'm happy to ...
1 vote
96 views

### Why KL divergence fails to approximate the means of distributions? [closed]

We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to ...
51 views

### Significance testing for Jensen–Shannon divergence?

The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions. How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
792 views

### KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler ...
28 views

### Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
45 views

### Quantifying if two datasets are from the same distribution, if I only have distances

Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample). I can compute the distance between instances. How do ...
116 views

### Does maximizing Jensen–Shannon divergence maximize Kullback–Leibler divergence?

Does maximizing the Jensen–Shannon divergence $D_{\mathrm{JS}}(P \parallel Q)$ maximize the Kullback–Leibler divergence $D_{\mathrm{KL}}(P \parallel Q)$? If so, I'd like to be able to show that it ...
57 views

### Is there a name for $\sum P(x) \frac{P(x)}{Q(x)}$ ? (P and Q are pmf)

I know that $\sum P(x) log \left( \frac{P(x)}{Q(x)} \right)$ is the kl-divergence. I'd like to know if there is a name for $\sum P(x) \left( \frac{P(x)}{Q(x)} \right)$ (no log), but couldn't find one. ...
1 vote
168 views

### KL divergence and Wasserstein distance

While reading the paper https://arxiv.org/pdf/1903.11780.pdf, I have some confusion parts as below: The KL divergence is not only problematic for representation learning due to the statistical ...
1 vote
29 views

### Proving that JSD is symmetric?

How can I prove that the Jensen–Shannon divergence (https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) is symmetric? Thanks!
24 views

### How to extend rank correlation to the continous case/infinite dimensions?

First let's assume that I want to compare two discrete distributions $d_1$ and $d_2$, but that I am not interested in their absolute values. Rather, I am interested in knowing how these two ...
30 views

1 vote
49 views

### Is there a rate of change performance measure for KL-divergence?

In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of ...
1 vote
237 views

### To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
1 vote
271 views

### Statistical distance between two matrices

The statistical distance between two probability distributions can be measured with $f$-divergences such as the KL-divergence. The statistical distance between two clusters can be measured with ...
183 views

### Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?

We know KL-divergence is sometimes expressed like this: which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
1 vote
1k views

### What is the correct way to implement Jensen-Shannon Distance?

I'm trying to use this code to compute the Jensen-Shannon distance: ...
373 views

### Sensitivity of KL Divergence

I am very new to the concept of KL divergence. Although I have grasped the fundamental formulations, I have a confusion comparing the KL divergence across the different distributions. Suppose I have 3 ...
607 views

### Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
141 views

### Is limiting density of discrete points (LDDP) equivalent to negative KL-divergence?

Is limiting density of discrete points (LDDP), which is a corrected version of differential entropy, equivalent to the negative KL-divergence (or relative entropy) between a density function $m(x)$ ...
270 views

### Is relative entropy equal to cross-entropy during optimization?

I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy ...
I have sequences of real-valued random variables $\{X_T\}, \{Y_T\}$ and a sequence of real numbers $\{a_T\}$. As $T\rightarrow\infty$, I know that $$a_T \rightarrow \infty$$ and  X_T \overset{d}{\...