Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

Filter by
Sorted by
Tagged with
1 vote
1 answer
20 views

Kl Divergence between factorized Gaussian and standard normal

Given two distributions, one a parameterized gaussian and the other a standard normal gaussian: $q(x) \sim \mathcal{N}(\mu,\sigma)$ $p(x) \sim \mathcal{N}(0,I)$ We want to compute the KL Divergence $...
user avatar
2 votes
0 answers
72 views

If two distributions have the same moments, how different can they be?

Let us suppose we have two distribution functions $F$ and $G$ with shared domain and also shared moments but not necessarily shared moment-generating functions. I have seen from "Whether ...
user avatar
  • 3,437
0 votes
0 answers
8 views

Significance of difference between empirical probability distributions

I have 1,000 pairs of probability distributions. Each pair $i$ consists of two discrete probability distributions that are measured empirically using $n_i$ and $m_i$ samples. i.e. every probability ...
user avatar
0 votes
0 answers
15 views

Analogous information matrix and divergence for the Bhattacharyya bound

In the case of Cramér-Rao lower bound (CRLB), the Fisher information matrix (FIM) is obtained from the K-L divergence (KLD), i.e. $D(p_\theta\|p_\theta') = \int p_\theta(x)\log\frac{p_\theta(x)}{p_{\...
user avatar
  • 153
0 votes
0 answers
6 views

What does the corresponding f(t) mean with respect to f-divergences?

In terms of f-divergences, what does the corresponding $f(t)$ refer to, what does it mean and why is it important? For example, with the KL-divergence the corresponding $f(t)$ is ${t \ log \ t}$. I ...
user avatar
  • 425
0 votes
0 answers
28 views

How to find the 'distance' between two populations?

I am somewhat new to these concepts, so please bear with me. I have two datasets: Data set A is collected by monitoring the network data of a device when it is ...
user avatar
  • 21
0 votes
0 answers
12 views

using divergence for comparing more than two data sets(100s) with varying number of data points

I have 100 companies and for each company, I have measured a feature of employees. The number of employees measured is not the same in all companies (the size of the companies can vary a lot). The ...
user avatar
  • 123
0 votes
0 answers
18 views

VAE divergence is positive in minimization of variational inference?

I have been going through the minimization of Variational inference and have a good understanding of all the steps taken: However, there is a part that relies on KL >= 0: I have derived the ...
user avatar
  • 3
0 votes
0 answers
11 views

Different notion of stationarity for distribution shift

Consider a stochastic process $(X_t)_{t\in T}$. Usually this will be a time series, so for simplicity consider $T=\mathbb{N}$ (or $T=\mathbb{Z}$). For many applications this process will be ...
user avatar
  • 105
0 votes
0 answers
10 views

Relating difference in distribution of dataset to information gain

I wonder if there are ways to relate difference in distribution of dataset to information gain. For example, I train a model on dataset D_1 and obtain a trained model M_1 and train a model on dataset ...
user avatar
  • 223
3 votes
1 answer
94 views

What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age). I would ...
user avatar
1 vote
1 answer
80 views

Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence: H(P, Q) = H(P) + D_KL(P||Q) Minimizing Cross-entropy means minimizing ...
user avatar
0 votes
0 answers
25 views

Measure for evaluating a density estimation procedure

Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure? I am currently evaluating the procedure using three ...
user avatar
  • 954
1 vote
1 answer
60 views

$\sqrt{2 KL(f || g)}$ interpretation?

I have seen in some papers that instead of using the Kullback-Leibler divergence $KL(f || g)$ between two probability density functions, $f$ and $g$, they use $$\sqrt{2 KL(f || g)}.$$ Is there any ...
user avatar
  • 11
4 votes
1 answer
58 views

Estimating the $\chi^2$-divergence with Monte Carlo: which distribution to sample from?

Notation: let the $\chi^2$-divergence between $p, q$ be defined as $$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x -1 = \int \frac{p(x)}{q(x)} p(x)\mathrm{d}x - 1. $$ ...
user avatar
  • 2,049
0 votes
0 answers
28 views

How do I interpolate a field that is divergence-free and curl-free at the same time?

A magnetic field is divergence free. At the points where there is no current, and no changing electric field, it is also curl free. There exist divergence-free and curl-free RBF kernels, and I could ...
user avatar
  • 101
1 vote
0 answers
34 views

Measuring the divergence in centrality statistics for similar networks?

I want to measure the similarity/divergence between the centrality of nodes in two publicly available word association networks. In my analysis, we have a long list of nodes - 12,000 or so - and then ...
user avatar
0 votes
0 answers
14 views

Grouping most strongly contributing items between different samples

I am performing an analysis by using Python's collections.Counter to obtain a scale of frequency x rank. I obtained the same statistics for multiple datasets and ...
user avatar
  • 1
2 votes
1 answer
80 views

Formal arguments for why an asymmetric f-divergence might be favourable to a symmetric one in analyzing importance sampling

I am reading Importance Sampling and Necessary Sample Size: an Information Theory Approach. Below is a quote from paragraph 3, section 3 of the article. While [total variation distance] and [...
user avatar
  • 2,049
1 vote
0 answers
80 views

Why is reverse KL more suited for data generation

Here goes a first question! In a paper I'm reading in the context of GAN's (WGAN in particular) I came across the following quote when the authors discuss KL divergence: while maximum likelihood ...
user avatar
0 votes
0 answers
33 views

Comparing two noisy probability measurements

I have two sampled probability values (sequence P = [p0, p1..pn] and sequence Q = [q0, q1,...qn]. Both of them are evaluated on time t0, t1...tn (equidistant). For simplicity, P is probability that ...
user avatar
3 votes
1 answer
553 views

What is meant by divergence in statistics?

I have learned about the Intuition on the Kullback-Leibler (KL) Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The two most important ...
user avatar
  • 2,391
1 vote
0 answers
29 views

estimate the divergence between two distributions by comparing the sufficient statistics

I'm interested in comparing two distributions $p(x,y)$ and $q(x,y)=pr(x)q(y|x)$. I want to estimate the KL-divergence between $p(x,y)$ and $q(x,y)$. It could be other divergence too, and I'm happy to ...
user avatar
1 vote
0 answers
96 views

Why KL divergence fails to approximate the means of distributions? [closed]

We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to ...
user avatar
0 votes
0 answers
51 views

Significance testing for Jensen–Shannon divergence?

The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions. How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
user avatar
4 votes
1 answer
792 views

KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler ...
user avatar
0 votes
0 answers
28 views

Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
user avatar
  • 279
0 votes
0 answers
45 views

Quantifying if two datasets are from the same distribution, if I only have distances

Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample). I can compute the distance between instances. How do ...
user avatar
4 votes
2 answers
116 views

Does maximizing Jensen–Shannon divergence maximize Kullback–Leibler divergence?

Does maximizing the Jensen–Shannon divergence $D_{\mathrm{JS}}(P \parallel Q)$ maximize the Kullback–Leibler divergence $D_{\mathrm{KL}}(P \parallel Q)$? If so, I'd like to be able to show that it ...
user avatar
2 votes
1 answer
57 views

Is there a name for $\sum P(x) \frac{P(x)}{Q(x)}$ ? (P and Q are pmf)

I know that $\sum P(x) log \left( \frac{P(x)}{Q(x)} \right)$ is the kl-divergence. I'd like to know if there is a name for $\sum P(x) \left( \frac{P(x)}{Q(x)} \right)$ (no log), but couldn't find one. ...
user avatar
1 vote
0 answers
168 views

KL divergence and Wasserstein distance

While reading the paper https://arxiv.org/pdf/1903.11780.pdf, I have some confusion parts as below: The KL divergence is not only problematic for representation learning due to the statistical ...
user avatar
  • 199
1 vote
1 answer
29 views

Proving that JSD is symmetric?

How can I prove that the Jensen–Shannon divergence (https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) is symmetric? Thanks!
user avatar
0 votes
0 answers
24 views

How to extend rank correlation to the continous case/infinite dimensions?

First let's assume that I want to compare two discrete distributions $d_1$ and $d_2$, but that I am not interested in their absolute values. Rather, I am interested in knowing how these two ...
user avatar
  • 239
0 votes
1 answer
30 views

Is the generalized entropy of certain events alway null?

In their paper: Novel Decompositions of Proper Scoring Rules for Classification, Kull and Flach wrote in section 2.2 Divergence, Entropy and Properness that when $y$ is the true class $d(p,y)=\phi(p,...
user avatar
  • 48
0 votes
1 answer
216 views

KL-divergence log likelihood ratio negative value

I have been trying to understand and implement KL-divergence for two normal distributions. However, one thing that I seem to be missing is how can KL-divergence always be a non-negative value, if the ...
user avatar
1 vote
0 answers
32 views

Is there a generalized form of the chi squared divergence?

Usually the chi squared divergence is used to compare two distributions. But is there a generalized form that also to compare k distributions? Intuitively I would consider to calculate the chi squared ...
user avatar
  • 31
1 vote
1 answer
2k views

What are the advantages of Wasserstein distance compared to Jensen-Shannon divergence?

What is the practical difference between Wasserstein metric and Jensen-Shannon divergence? Wasserstein metric is also referred to as Earth mover's distance. From Wikipedia: Wasserstein metric is a ...
user avatar
  • 3,047
2 votes
0 answers
28 views

Strong data processing inequality in multiplicative channels

We know that postprocessing will not increase the information. For two random variables $X$ and $Y$, $D(X||Y)>= D(f(X)||f(Y))$ for any operation $f()$ and divergence $D$. A strong data processing ...
user avatar
  • 21
3 votes
1 answer
2k views

Optimizing forward/reverse KL divergence for Gaussian distributions

The forward/reverse formulations of KL divergence are distinguished by having mean/mode-seeking behavior. The typical example for using KL to optimize a distribution $Q_\theta$ to fit a distribution $...
user avatar
1 vote
0 answers
49 views

Is there a rate of change performance measure for KL-divergence?

In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of ...
user avatar
  • 3,047
1 vote
1 answer
237 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
user avatar
  • 3,047
1 vote
1 answer
271 views

Statistical distance between two matrices

The statistical distance between two probability distributions can be measured with $f$-divergences such as the KL-divergence. The statistical distance between two clusters can be measured with ...
user avatar
  • 3,047
2 votes
0 answers
183 views

Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?

We know KL-divergence is sometimes expressed like this: which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
user avatar
1 vote
3 answers
1k views

What is the correct way to implement Jensen-Shannon Distance?

I'm trying to use this code to compute the Jensen-Shannon distance: ...
user avatar
2 votes
1 answer
373 views

Sensitivity of KL Divergence

I am very new to the concept of KL divergence. Although I have grasped the fundamental formulations, I have a confusion comparing the KL divergence across the different distributions. Suppose I have 3 ...
user avatar
2 votes
1 answer
607 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
user avatar
  • 3,047
3 votes
1 answer
141 views

Is limiting density of discrete points (LDDP) equivalent to negative KL-divergence?

Is limiting density of discrete points (LDDP), which is a corrected version of differential entropy, equivalent to the negative KL-divergence (or relative entropy) between a density function $m(x)$ ...
user avatar
  • 3,047
4 votes
1 answer
270 views

Is relative entropy equal to cross-entropy during optimization?

I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy ...
user avatar
  • 3,047
2 votes
0 answers
133 views

Show that a sequence of random variables diverges to infinity in probability

I have sequences of real-valued random variables $\{X_T\}, \{Y_T\}$ and a sequence of real numbers $\{a_T\}$. As $T\rightarrow\infty$, I know that $$ a_T \rightarrow \infty $$ and $$ X_T \overset{d}{\...
user avatar
  • 83
1 vote
0 answers
31 views

What are the best known techniques to verify that a GAN samples correctly from a given distribution?

I would like to know what are the best known techniques to check that a generative adversarial network (GAN) samples from the correct distribution. Naively, I would say it all boils down to a ...
user avatar
  • 3,888