# Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

70 questions
Filter by
Sorted by
Tagged with
15 views

### Generalised Jensen-Shannon Divergence - Unequal Length Probability Distributions

I'd like to implement a generalized Jensen-Shannon divergence (GJSD) style test comparing 3 different probability distributions. In this respect, I looked at the Philentropy library in R with the ...
• 244
62 views

### Negative KL Divergence estimates

I was exploring the KL Divergence and came across some research about calculating it from samples. On stack-exchange, I found out that minimising the KL Divergence is equivalent to minimising the Sum ...
• 21
67 views

### Is this a known or valid divergence between two densities?

I am testing various metrics for learning a density estimate. Specifically, I have a sample of data from a distribution $p$, and am learning a function $f$ to estimate $p$ by minimizing a distance or ...
• 181
82 views

### Z-test with no relevant sample size as I have a Gaussian probability distribution

I have two Gaussian curves, there are not samples, these are just probability distributions essentially. So I can do a Gaussian fit on them, or also a weighted average and weighted variance on the ...
28 views

### What are distances, metrics, or divergences to determine if one sample comes from components of a distribution

I have a distribution that, wlog, can be defined as a mixture of component distributions $\mathscr{D} = \alpha_{1}\mathscr{D}_{1} + \alpha_{2}\mathscr{D}_{2} + \dots$. Is there a metric, distance, or ...
132 views

### Is there sampling process that admits computing a similarity of two densities when one is intractable?

I have two densities, $p, q$ with sample space $\mathbb{R}^n$, and we can assume both $p,q>0$ (full support). I can compute and sample from $q$. I can compute $p$ up to a constant and I cannot ...
19 views

### Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?

I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
27 views

### How to measure the difference between two distributions of the same family?

Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
1 vote
648 views

### How is Jensen–Shannon divergence bounded between [0,1]?

According to the Wiki, Jensen–Shannon divergence (JSD) is bounded between [0,1]. I am having trouble understanding why this is. Let's say $p_1 ~ N(\mu_1,\sigma^2)$, and $p_2 ~ N(\mu_2,\sigma^2)$ (same ...
90 views

### How do I measure the "dispersions" of a group of time series

I have a group of time series $X_1, X_2, ... X_n$. I want to measure how much they have "dispersed" over time. i.e. are they moving "more together" in 2023, comparing to 2022. $n$ ...
• 256
56 views

### Apply divergences between two "relative frequency distributions", instead of between two "probability distributions"

Introduction. Recalling that: The frequency is the number of observations of a specific outcome. The relative frequency is a proportion of all observations (frequency / total observations). A "...
• 280
74 views

### Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
3k views

### How do you find the KL Divergence between two multi-variable datasets?

Background I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
• 655
44 views

### Sample from one distribution such that it’s PDF matches another distribution

Problem: I have a set of samples from a continuous distribution (multivariate), call this set $W$. I have another set of samples from a different distribution $X$. I want to sample from $W$ (with ...
1 vote
85 views

### In MAP, does maximizing the posterior minimize any divergence between distributions?

It's known that maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler divergence between the model $q(x \mid \theta)$ and the unknown true data generating distribution $p(x)$: ...
• 330
1 vote
162 views

### How do I compare multivariate normal distributions and get a p-value?

I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations). How do I quantify the distance (or ...
54 views

### Measuring distance between two continuous distributions using their discrete approximations

I need to compute the distance between two continuous distributions. However, I have no idea as to what kind of distributions they are. I have a discrete approximation of the distributions. That is, ...
2k views

### Kullback–Leibler divergence between two normal distributions

I am currently reading 'Dive into Deep Learning' and right now I am trying to improve my intuition for the Kullback–Leibler divergence. I get the basic idea, why this metric is not symmetric, however, ...
• 535
248 views

### How to derive the Jensen-Shannon divergence from the f-divergence?

The Jensen-Shannon divergence is defined as $$JS(p, q) = \frac{1}{2}\left(KL\left(p||\frac{p+q}{2}\right) + KL\left(q||\frac{p+q}{2}\right) \right).$$ In Wikipedia it says that it can be derived from ...
410 views

### Understanding KL divergence in chapter 7 of Statistical Rethinking

I'm having a hard time understanding McElreath's explanation of how the KL divergence allows us to decide whether one of two models is closer to the 'real' model. Here is what McElreath writes on p. ...
• 551
64 views

### Is there a standard name for this variant of KL divergence?

Since KL divergence can be decomposed as \begin{equation*} D_{\mathrm{KL}}(q \| p) = H(p \| q) - H(p), \end{equation*} I wonder if there exists a weighted version of KL divergence, i.e., \begin{...
• 143
1 vote
91 views

### KL divergence and the MAP approximation in BNNs

I was reading this blog post on bayesian neural networks, where the author shows that if we use as a variational distribution a product of delta function, then minimizing the loss function of a BNN is ...
• 325
33 views

### Does this similarity measure have a name?

Consider two probability distributions P and Q defined on the same probability space X. Does the following similarity measure have a name? Context: I "invented" this formula to compare ...
• 787
151 views

### Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
• 433
1 vote