Questions tagged [divergence]
a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.
66
questions
0
votes
0
answers
14
views
Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?
I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
- 31
0
votes
0
answers
24
views
How to measure the difference between two distributions of the same family?
Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
1
vote
1
answer
129
views
How is Jensen–Shannon divergence bounded between [0,1]?
According to the Wiki, Jensen–Shannon divergence (JSD) is bounded between [0,1]. I am having trouble understanding why this is.
Let's say $p_1 ~ N(\mu_1,\sigma^2)$, and $p_2 ~ N(\mu_2,\sigma^2)$ (same ...
- 111
0
votes
0
answers
54
views
How do I measure the "dispersions" of a group of time series
I have a group of time series $X_1, X_2, ... X_n$. I want to measure how much they have "dispersed" over time. i.e. are they moving "more together" in 2023, comparing to 2022. $n$ ...
- 133
-1
votes
1
answer
50
views
Apply divergences between two "relative frequency distributions", instead of between two "probability distributions"
Introduction. Recalling that:
The frequency is the number of observations of a specific outcome.
The relative frequency is a proportion of all observations (frequency / total observations).
A "...
- 168
0
votes
0
answers
44
views
KL divergence for disjoint distributions
According to the article here, we have two disjoint distributions as shown below.
.
KL divergence for the distributions are
I don't understand why denominators are 0 for both.
- 483
0
votes
0
answers
91
views
Does Sliced Wasserstein Distance assume that the two distributions are zero-mean?
Sliced Wasserstein distance (SWD) depends on Radon transform, which is defined as $\mathbb{R}I(t,\theta) = \int \delta(t-\theta^Tx)I(x) \, dx$. The definition of SWD, however, does not use the ...
- 236
0
votes
0
answers
52
views
Which metric to compare two probability density?
I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
- 1
5
votes
2
answers
2k
views
How do you find the KL Divergence between two multi-variable datasets?
Background
I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
- 411
0
votes
0
answers
41
views
Sample from one distribution such that it’s PDF matches another distribution
Problem: I have a set of samples from a continuous distribution (multivariate), call this set $W$. I have another set of samples from a different distribution $X$. I want to sample from $W$ (with ...
- 21
1
vote
0
answers
65
views
In MAP, does maximizing the posterior minimize any divergence between distributions?
It's known that maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler divergence between the model $q(x \mid \theta)$ and the unknown true data generating distribution $p(x)$:
...
- 310
1
vote
1
answer
110
views
How do I compare multivariate normal distributions and get a p-value?
I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations).
How do I quantify the distance (or ...
- 13
2
votes
0
answers
47
views
Measuring distance between two continuous distributions using their discrete approximations
I need to compute the distance between two continuous distributions. However, I have no idea as to what kind of distributions they are. I have a discrete approximation of the distributions. That is, ...
- 149
5
votes
1
answer
1k
views
Kullback–Leibler divergence between two normal distributions
I am currently reading 'Dive into Deep Learning' and right now I am trying to improve my intuition for the Kullback–Leibler divergence. I get the basic idea, why this metric is not symmetric, however, ...
- 505
1
vote
1
answer
161
views
How to derive the Jensen-Shannon divergence from the f-divergence?
The Jensen-Shannon divergence is defined as $$JS(p, q) = \frac{1}{2}\left(KL\left(p||\frac{p+q}{2}\right) + KL\left(q||\frac{p+q}{2}\right) \right).$$ In Wikipedia it says that it can be derived from ...
3
votes
1
answer
310
views
Understanding KL divergence in chapter 7 of Statistical Rethinking
I'm having a hard time understanding McElreath's explanation of how the KL divergence allows us to decide whether one of two models is closer to the 'real' model.
Here is what McElreath writes on p. ...
- 541
2
votes
0
answers
60
views
Is there a standard name for this variant of KL divergence?
Since KL divergence can be decomposed as
\begin{equation*}
D_{\mathrm{KL}}(q \| p) = H(p \| q) - H(p),
\end{equation*}
I wonder if there exists a weighted version of KL divergence, i.e.,
\begin{...
- 143
1
vote
0
answers
76
views
KL divergence and the MAP approximation in BNNs
I was reading this blog post on bayesian neural networks, where the author shows that if we use as a variational distribution a product of delta function, then minimizing the loss function of a BNN is ...
- 325
0
votes
1
answer
33
views
Does this similarity measure have a name?
Consider two probability distributions P and Q defined on the same probability space X. Does the following similarity measure have a name?
Context: I "invented" this formula to compare ...
- 787
2
votes
1
answer
109
views
Why is it called the cross-entropy of q relative to p, not p relative to q?
I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy
Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
- 423
1
vote
1
answer
212
views
Kl Divergence between factorized Gaussian and standard normal
Given two distributions, one a parameterized gaussian and the other a standard normal gaussian:
$q(x) \sim \mathcal{N}(\mu,\sigma)$
$p(x) \sim \mathcal{N}(0,I)$
We want to compute the KL Divergence $...
3
votes
1
answer
540
views
If two distributions have the same moments, how different can they be?
Let us suppose we have two distribution functions $F$ and $G$ with shared domain and also shared moments but not necessarily shared moment-generating functions.
I have seen from "Whether ...
- 7,118
1
vote
0
answers
50
views
How to find the 'distance' between two populations?
I am somewhat new to these concepts, so please bear with me.
I have two datasets: Data set A is collected by monitoring the network data of a device when it is ...
- 33
0
votes
0
answers
74
views
VAE divergence is positive in minimization of variational inference?
I have been going through the minimization of Variational inference and have a good understanding of all the steps taken:
However, there is a part that relies on KL >= 0:
I have derived the ...
- 3
5
votes
1
answer
520
views
What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)
I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age).
I would ...
- 21.1k
1
vote
1
answer
542
views
Why KL divergence close to zero when Q close to P?
I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence:
H(P, Q) = H(P) + D_KL(P||Q)
Minimizing Cross-entropy means minimizing ...
0
votes
0
answers
100
views
Measure for evaluating a density estimation procedure
Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure?
I am currently evaluating the procedure using three ...
- 2,646
3
votes
1
answer
104
views
$\sqrt{2 KL(f || g)}$ interpretation?
I have seen in some papers that instead of using the Kullback-Leibler divergence $KL(f || g)$ between two probability density functions, $f$ and $g$, they use
$$\sqrt{2 KL(f || g)}.$$
Is there any ...
- 31
5
votes
1
answer
67
views
Estimating the $\chi^2$-divergence with Monte Carlo: which distribution to sample from?
Notation: let the $\chi^2$-divergence between $p, q$ be defined as
$$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x -1 = \int \frac{p(x)}{q(x)} p(x)\mathrm{d}x - 1. $$
...
- 2,416
0
votes
0
answers
78
views
How do I interpolate a field that is divergence-free and curl-free at the same time?
A magnetic field is divergence free. At the points where there is no current, and no changing electric field, it is also curl free.
There exist divergence-free and curl-free RBF kernels, and I could ...
- 131
1
vote
0
answers
38
views
Measuring the divergence in centrality statistics for similar networks?
I want to measure the similarity/divergence between the centrality of nodes in two publicly available word association networks.
In my analysis, we have a long list of nodes - 12,000 or so - and then ...
3
votes
1
answer
214
views
Formal arguments for why an asymmetric f-divergence might be favourable to a symmetric one in analyzing importance sampling
I am reading Importance Sampling and Necessary Sample Size: an Information Theory Approach.
Below is a quote from paragraph 3, section 3 of the article.
While [total variation distance] and [...
- 2,416
1
vote
0
answers
336
views
Why is reverse KL more suited for data generation
Here goes a first question! In a paper I'm reading in the context of GAN's (WGAN in particular) I came across the following quote when the authors discuss KL divergence:
while maximum likelihood ...
- 11
0
votes
0
answers
38
views
Comparing two noisy probability measurements
I have two sampled probability values (sequence P = [p0, p1..pn] and sequence Q = [q0, q1,...qn]. Both of them are evaluated on time t0, t1...tn (equidistant). For simplicity, P is probability that ...
4
votes
1
answer
3k
views
What is meant by divergence in statistics?
I have learned about the Intuition on the Kullback-Leibler (KL) Divergence as how much a model distribution function differs from the theoretical/true distribution of the data.
The two most important ...
- 2,785
1
vote
0
answers
33
views
estimate the divergence between two distributions by comparing the sufficient statistics [closed]
I'm interested in comparing two distributions $p(x,y)$ and $q(x,y)=pr(x)q(y|x)$.
I want to estimate the KL-divergence between $p(x,y)$ and $q(x,y)$. It could be other divergence too, and I'm happy to ...
- 111
1
vote
0
answers
207
views
Why KL divergence fails to approximate the means of distributions? [closed]
We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to ...
- 212
0
votes
0
answers
297
views
Significance testing for Jensen–Shannon divergence?
The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions.
How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
- 2,932
7
votes
1
answer
2k
views
KL divergence for joint probability distributions?
I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler ...
- 2,932
0
votes
0
answers
77
views
Cross entropy error: Poor modelling giving too much weight to unlikely events
I was reading this paper. link (page 5)
In this paper, there is a statement that goes like this:
To begin, cross entropy error is just one among many possible distance
measures between probability ...
- 279
0
votes
0
answers
154
views
Quantifying if two datasets are from the same distribution, if I only have distances
Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample).
I can compute the distance between instances.
How do ...
- 101
4
votes
2
answers
329
views
Does maximizing Jensen–Shannon divergence maximize Kullback–Leibler divergence?
Does maximizing the Jensen–Shannon divergence $D_{\mathrm{JS}}(P \parallel Q)$ maximize the Kullback–Leibler divergence $D_{\mathrm{KL}}(P \parallel Q)$? If so, I'd like to be able to show that it ...
- 41
3
votes
1
answer
69
views
Is there a name for $\sum P(x) \frac{P(x)}{Q(x)}$ ? (P and Q are pmf)
I know that $\sum P(x) log \left( \frac{P(x)}{Q(x)} \right)$ is the kl-divergence. I'd like to know if there is a name for $\sum P(x) \left( \frac{P(x)}{Q(x)} \right)$ (no log), but couldn't find one.
...
- 21.1k
1
vote
0
answers
639
views
KL divergence and Wasserstein distance
While reading the paper https://arxiv.org/pdf/1903.11780.pdf, I have some confusion parts as below:
The KL divergence is not only problematic for representation learning
due to the statistical ...
- 251
1
vote
1
answer
137
views
Proving that JSD is symmetric?
How can I prove that the Jensen–Shannon divergence
(https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence)
is symmetric?
Thanks!
- 19
0
votes
0
answers
30
views
How to extend rank correlation to the continous case/infinite dimensions?
First let's assume that I want to compare two discrete distributions $d_1$ and $d_2$, but that I am not interested in their absolute values. Rather, I am interested in knowing how these two ...
- 239
0
votes
1
answer
43
views
Is the generalized entropy of certain events alway null?
In their paper: Novel Decompositions of Proper Scoring Rules for Classification, Kull and Flach wrote in section 2.2 Divergence, Entropy and Properness that when $y$ is the true class $d(p,y)=\phi(p,...
- 148
0
votes
1
answer
820
views
KL-divergence log likelihood ratio negative value
I have been trying to understand and implement KL-divergence for two normal distributions. However, one thing that I seem to be missing is how can KL-divergence always be a non-negative value, if the ...
1
vote
0
answers
72
views
Is there a generalized form of the chi squared divergence?
Usually the chi squared divergence is used to compare two distributions. But is there a generalized form that also to compare k distributions?
Intuitively I would consider to calculate the chi squared ...
- 41
2
votes
1
answer
4k
views
What are the advantages of Wasserstein distance compared to Jensen-Shannon divergence?
What is the practical difference between Wasserstein metric and Jensen-Shannon divergence? Wasserstein metric is also referred to as Earth mover's distance.
From Wikipedia:
Wasserstein metric is a ...
- 3,559