Skip to main content

Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

Generalised Jensen-Shannon Divergence - Unequal Length Probability Distributions

I'd like to implement a generalized Jensen-Shannon divergence (GJSD) style test comparing 3 different probability distributions. In this respect, I looked at the Philentropy library in R with the ...
EB3112's user avatar
  • 244
2 votes
0 answers
62 views

Negative KL Divergence estimates

I was exploring the KL Divergence and came across some research about calculating it from samples. On stack-exchange, I found out that minimising the KL Divergence is equivalent to minimising the Sum ...
Beetel's user avatar
  • 21
3 votes
1 answer
67 views

Is this a known or valid divergence between two densities?

I am testing various metrics for learning a density estimate. Specifically, I have a sample of data from a distribution $p$, and am learning a function $f$ to estimate $p$ by minimizing a distance or ...
Travis L's user avatar
  • 181
2 votes
1 answer
82 views

Z-test with no relevant sample size as I have a Gaussian probability distribution

I have two Gaussian curves, there are not samples, these are just probability distributions essentially. So I can do a Gaussian fit on them, or also a weighted average and weighted variance on the ...
Dominik Duleba's user avatar
0 votes
0 answers
28 views

What are distances, metrics, or divergences to determine if one sample comes from components of a distribution

I have a distribution that, wlog, can be defined as a mixture of component distributions $\mathscr{D} = \alpha_{1}\mathscr{D}_{1} + \alpha_{2}\mathscr{D}_{2} + \dots$. Is there a metric, distance, or ...
Alex Hagen's user avatar
6 votes
2 answers
132 views

Is there sampling process that admits computing a similarity of two densities when one is intractable?

I have two densities, $p, q$ with sample space $\mathbb{R}^n$, and we can assume both $p,q>0$ (full support). I can compute and sample from $q$. I can compute $p$ up to a constant and I cannot ...
travelingbones's user avatar
0 votes
0 answers
19 views

Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?

I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
user17420392's user avatar
0 votes
0 answers
27 views

How to measure the difference between two distributions of the same family?

Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
Value_Investor's user avatar
1 vote
1 answer
648 views

How is Jensen–Shannon divergence bounded between [0,1]?

According to the Wiki, Jensen–Shannon divergence (JSD) is bounded between [0,1]. I am having trouble understanding why this is. Let's say $p_1 ~ N(\mu_1,\sigma^2)$, and $p_2 ~ N(\mu_2,\sigma^2)$ (same ...
lowlyprogrammer's user avatar
0 votes
0 answers
90 views

How do I measure the "dispersions" of a group of time series

I have a group of time series $X_1, X_2, ... X_n$. I want to measure how much they have "dispersed" over time. i.e. are they moving "more together" in 2023, comparing to 2022. $n$ ...
Taylor Fang's user avatar
-1 votes
1 answer
56 views

Apply divergences between two "relative frequency distributions", instead of between two "probability distributions"

Introduction. Recalling that: The frequency is the number of observations of a specific outcome. The relative frequency is a proportion of all observations (frequency / total observations). A "...
Ommo's user avatar
  • 280
0 votes
0 answers
74 views

Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
YP BARRY's user avatar
5 votes
2 answers
3k views

How do you find the KL Divergence between two multi-variable datasets?

Background I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
Connor's user avatar
  • 655
0 votes
0 answers
44 views

Sample from one distribution such that it’s PDF matches another distribution

Problem: I have a set of samples from a continuous distribution (multivariate), call this set $W$. I have another set of samples from a different distribution $X$. I want to sample from $W$ (with ...
user102938's user avatar
1 vote
0 answers
85 views

In MAP, does maximizing the posterior minimize any divergence between distributions?

It's known that maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler divergence between the model $q(x \mid \theta)$ and the unknown true data generating distribution $p(x)$: ...
ForceBru's user avatar
  • 330
1 vote
1 answer
162 views

How do I compare multivariate normal distributions and get a p-value?

I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations). How do I quantify the distance (or ...
Sam Huguet's user avatar
2 votes
0 answers
54 views

Measuring distance between two continuous distributions using their discrete approximations

I need to compute the distance between two continuous distributions. However, I have no idea as to what kind of distributions they are. I have a discrete approximation of the distributions. That is, ...
Nagabhushan S N's user avatar
5 votes
1 answer
2k views

Kullback–Leibler divergence between two normal distributions

I am currently reading 'Dive into Deep Learning' and right now I am trying to improve my intuition for the Kullback–Leibler divergence. I get the basic idea, why this metric is not symmetric, however, ...
kklaw's user avatar
  • 535
2 votes
1 answer
248 views

How to derive the Jensen-Shannon divergence from the f-divergence?

The Jensen-Shannon divergence is defined as $$JS(p, q) = \frac{1}{2}\left(KL\left(p||\frac{p+q}{2}\right) + KL\left(q||\frac{p+q}{2}\right) \right).$$ In Wikipedia it says that it can be derived from ...
tintinnabulum's user avatar
3 votes
1 answer
410 views

Understanding KL divergence in chapter 7 of Statistical Rethinking

I'm having a hard time understanding McElreath's explanation of how the KL divergence allows us to decide whether one of two models is closer to the 'real' model. Here is what McElreath writes on p. ...
matsuo_basho's user avatar
2 votes
0 answers
64 views

Is there a standard name for this variant of KL divergence?

Since KL divergence can be decomposed as \begin{equation*} D_{\mathrm{KL}}(q \| p) = H(p \| q) - H(p), \end{equation*} I wonder if there exists a weighted version of KL divergence, i.e., \begin{...
Ze-Nan Li's user avatar
  • 143
1 vote
0 answers
91 views

KL divergence and the MAP approximation in BNNs

I was reading this blog post on bayesian neural networks, where the author shows that if we use as a variational distribution a product of delta function, then minimizing the loss function of a BNN is ...
Alucard's user avatar
  • 325
0 votes
1 answer
33 views

Does this similarity measure have a name?

Consider two probability distributions P and Q defined on the same probability space X. Does the following similarity measure have a name? Context: I "invented" this formula to compare ...
KaPy3141's user avatar
  • 787
2 votes
1 answer
151 views

Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
user900476's user avatar
1 vote
1 answer
396 views

Kl Divergence between factorized Gaussian and standard normal

Given two distributions, one a parameterized gaussian and the other a standard normal gaussian: $q(x) \sim \mathcal{N}(\mu,\sigma)$ $p(x) \sim \mathcal{N}(0,I)$ We want to compute the KL Divergence $...
Martin Bucher's user avatar
3 votes
1 answer
650 views

If two distributions have the same moments, how different can they be?

Let us suppose we have two distribution functions $F$ and $G$ with shared domain and also shared moments but not necessarily shared moment-generating functions. I have seen from "Whether ...
Galen's user avatar
  • 9,381
1 vote
0 answers
56 views

How to find the 'distance' between two populations?

I am somewhat new to these concepts, so please bear with me. I have two datasets: Data set A is collected by monitoring the network data of a device when it is ...
Nht_e0's user avatar
  • 33
0 votes
0 answers
81 views

VAE divergence is positive in minimization of variational inference?

I have been going through the minimization of Variational inference and have a good understanding of all the steps taken: However, there is a part that relies on KL >= 0: I have derived the ...
Frank's user avatar
  • 3
7 votes
1 answer
804 views

What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)

I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age). I would ...
Tal Galili's user avatar
  • 21.8k
1 vote
1 answer
683 views

Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence: H(P, Q) = H(P) + D_KL(P||Q) Minimizing Cross-entropy means minimizing ...
Nisan Chhetri's user avatar
0 votes
0 answers
117 views

Measure for evaluating a density estimation procedure

Given an implementation of a multivariate density estimation scheme, what would be a suitable measure to evaluate the accuracy of the procedure? I am currently evaluating the procedure using three ...
Eike P.'s user avatar
  • 3,118
3 votes
1 answer
122 views

$\sqrt{2 KL(f || g)}$ interpretation?

I have seen in some papers that instead of using the Kullback-Leibler divergence $KL(f || g)$ between two probability density functions, $f$ and $g$, they use $$\sqrt{2 KL(f || g)}.$$ Is there any ...
Leon-o's user avatar
  • 31
5 votes
1 answer
67 views

Estimating the $\chi^2$-divergence with Monte Carlo: which distribution to sample from?

Notation: let the $\chi^2$-divergence between $p, q$ be defined as $$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x -1 = \int \frac{p(x)}{q(x)} p(x)\mathrm{d}x - 1. $$ ...
fool's user avatar
  • 2,480
0 votes
0 answers
94 views

How do I interpolate a field that is divergence-free and curl-free at the same time?

A magnetic field is divergence free. At the points where there is no current, and no changing electric field, it is also curl free. There exist divergence-free and curl-free RBF kernels, and I could ...
seed's user avatar
  • 131
1 vote
0 answers
39 views

Measuring the divergence in centrality statistics for similar networks?

I want to measure the similarity/divergence between the centrality of nodes in two publicly available word association networks. In my analysis, we have a long list of nodes - 12,000 or so - and then ...
Peter Thwaites's user avatar
3 votes
1 answer
256 views

Formal arguments for why an asymmetric f-divergence might be favourable to a symmetric one in analyzing importance sampling

I am reading Importance Sampling and Necessary Sample Size: an Information Theory Approach. Below is a quote from paragraph 3, section 3 of the article. While [total variation distance] and [...
fool's user avatar
  • 2,480
1 vote
0 answers
407 views

Why is reverse KL more suited for data generation

Here goes a first question! In a paper I'm reading in the context of GAN's (WGAN in particular) I came across the following quote when the authors discuss KL divergence: while maximum likelihood ...
Niels Ota's user avatar
0 votes
0 answers
38 views

Comparing two noisy probability measurements

I have two sampled probability values (sequence P = [p0, p1..pn] and sequence Q = [q0, q1,...qn]. Both of them are evaluated on time t0, t1...tn (equidistant). For simplicity, P is probability that ...
mangled data's user avatar
4 votes
1 answer
3k views

What is meant by divergence in statistics?

I have learned about the Intuition on the Kullback-Leibler (KL) Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The two most important ...
Thalassophile's user avatar
1 vote
0 answers
33 views

estimate the divergence between two distributions by comparing the sufficient statistics [closed]

I'm interested in comparing two distributions $p(x,y)$ and $q(x,y)=pr(x)q(y|x)$. I want to estimate the KL-divergence between $p(x,y)$ and $q(x,y)$. It could be other divergence too, and I'm happy to ...
Jiaji Huang's user avatar
1 vote
0 answers
223 views

Why KL divergence fails to approximate the means of distributions? [closed]

We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to ...
Kirk Walla's user avatar
0 votes
0 answers
467 views

Significance testing for Jensen–Shannon divergence?

The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions. How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
rhombidodecahedron's user avatar
7 votes
1 answer
2k views

KL divergence for joint probability distributions?

I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the Kullback–Leibler ...
rhombidodecahedron's user avatar
0 votes
0 answers
88 views

Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
user27286's user avatar
  • 299
0 votes
0 answers
161 views

Quantifying if two datasets are from the same distribution, if I only have distances

Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample). I can compute the distance between instances. How do ...
Joseph Turian's user avatar
4 votes
2 answers
400 views

Does maximizing Jensen–Shannon divergence maximize Kullback–Leibler divergence?

Does maximizing the Jensen–Shannon divergence $D_{\mathrm{JS}}(P \parallel Q)$ maximize the Kullback–Leibler divergence $D_{\mathrm{KL}}(P \parallel Q)$? If so, I'd like to be able to show that it ...
seabass09's user avatar
3 votes
1 answer
70 views

Is there a name for $\sum P(x) \frac{P(x)}{Q(x)}$ ? (P and Q are pmf)

I know that $\sum P(x) log \left( \frac{P(x)}{Q(x)} \right)$ is the kl-divergence. I'd like to know if there is a name for $\sum P(x) \left( \frac{P(x)}{Q(x)} \right)$ (no log), but couldn't find one. ...
Tal Galili's user avatar
  • 21.8k
1 vote
0 answers
765 views

KL divergence and Wasserstein distance

While reading the paper https://arxiv.org/pdf/1903.11780.pdf, I have some confusion parts as below: The KL divergence is not only problematic for representation learning due to the statistical ...
alryosha's user avatar
  • 271
1 vote
1 answer
159 views

Proving that JSD is symmetric?

How can I prove that the Jensen–Shannon divergence (https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) is symmetric? Thanks!
user309781's user avatar
0 votes
0 answers
30 views

How to extend rank correlation to the continous case/infinite dimensions?

First let's assume that I want to compare two discrete distributions $d_1$ and $d_2$, but that I am not interested in their absolute values. Rather, I am interested in knowing how these two ...
Ash's user avatar
  • 239