# Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

66 questions
Filter by
Sorted by
Tagged with
14 views

### Are there any research papers which show why Wasserstein distance is better than Jensen-Shannon/KL_div/Bhattacharya distance for specific use cases?

I am trying to find reliable research work which show why displacement based metrics such as Wasserstein distance is a better suited metric than Jensen-Shannon distance in specific use cases and for ...
24 views

### How to measure the difference between two distributions of the same family?

Kullback-Leibler divergence seems to be a frequently used "metric" to measure the difference between probability distributions, regardless of their respective families. However, I would like ...
1 vote
129 views

### How is Jensen–Shannon divergence bounded between [0,1]?

According to the Wiki, Jensen–Shannon divergence (JSD) is bounded between [0,1]. I am having trouble understanding why this is. Let's say $p_1 ~ N(\mu_1,\sigma^2)$, and $p_2 ~ N(\mu_2,\sigma^2)$ (same ...
54 views

### How do I measure the "dispersions" of a group of time series

I have a group of time series $X_1, X_2, ... X_n$. I want to measure how much they have "dispersed" over time. i.e. are they moving "more together" in 2023, comparing to 2022. $n$ ...
50 views

### Apply divergences between two "relative frequency distributions", instead of between two "probability distributions"

Introduction. Recalling that: The frequency is the number of observations of a specific outcome. The relative frequency is a proportion of all observations (frequency / total observations). A "...
44 views

### KL divergence for disjoint distributions

According to the article here, we have two disjoint distributions as shown below. . KL divergence for the distributions are I don't understand why denominators are 0 for both.
91 views

### Does Sliced Wasserstein Distance assume that the two distributions are zero-mean?

Sliced Wasserstein distance (SWD) depends on Radon transform, which is defined as $\mathbb{R}I(t,\theta) = \int \delta(t-\theta^Tx)I(x) \, dx$. The definition of SWD, however, does not use the ...
52 views

### Which metric to compare two probability density?

I need to compare two distribution $p$ and $q$. But I don't have access to the distribution $p$, I want to approximate it by distribution $q$ that I construct iteratively by choosing design point. ...
2k views

### How do you find the KL Divergence between two multi-variable datasets?

Background I'm working on a tabular data model that performs a binary classification. The model has recently started underperforming and I'd like to know if that's due to a drift in the feature ...
41 views

### Sample from one distribution such that it’s PDF matches another distribution

Problem: I have a set of samples from a continuous distribution (multivariate), call this set $W$. I have another set of samples from a different distribution $X$. I want to sample from $W$ (with ...
1 vote
65 views

### In MAP, does maximizing the posterior minimize any divergence between distributions?

It's known that maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler divergence between the model $q(x \mid \theta)$ and the unknown true data generating distribution $p(x)$: ...
1 vote
110 views

### How do I compare multivariate normal distributions and get a p-value?

I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations). How do I quantify the distance (or ...
47 views

### Measuring distance between two continuous distributions using their discrete approximations

I need to compute the distance between two continuous distributions. However, I have no idea as to what kind of distributions they are. I have a discrete approximation of the distributions. That is, ...
1k views

### Kullback–Leibler divergence between two normal distributions

I am currently reading 'Dive into Deep Learning' and right now I am trying to improve my intuition for the Kullback–Leibler divergence. I get the basic idea, why this metric is not symmetric, however, ...
1 vote
161 views

### How to derive the Jensen-Shannon divergence from the f-divergence?

The Jensen-Shannon divergence is defined as $$JS(p, q) = \frac{1}{2}\left(KL\left(p||\frac{p+q}{2}\right) + KL\left(q||\frac{p+q}{2}\right) \right).$$ In Wikipedia it says that it can be derived from ...
310 views

### Understanding KL divergence in chapter 7 of Statistical Rethinking

I'm having a hard time understanding McElreath's explanation of how the KL divergence allows us to decide whether one of two models is closer to the 'real' model. Here is what McElreath writes on p. ...
60 views

### Is there a standard name for this variant of KL divergence?

Since KL divergence can be decomposed as \begin{equation*} D_{\mathrm{KL}}(q \| p) = H(p \| q) - H(p), \end{equation*} I wonder if there exists a weighted version of KL divergence, i.e., \begin{...
1 vote
76 views

### KL divergence and the MAP approximation in BNNs

I was reading this blog post on bayesian neural networks, where the author shows that if we use as a variational distribution a product of delta function, then minimizing the loss function of a BNN is ...
33 views

### Does this similarity measure have a name?

Consider two probability distributions P and Q defined on the same probability space X. Does the following similarity measure have a name? Context: I "invented" this formula to compare ...
109 views

### Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
1 vote
212 views

820 views

### KL-divergence log likelihood ratio negative value

I have been trying to understand and implement KL-divergence for two normal distributions. However, one thing that I seem to be missing is how can KL-divergence always be a non-negative value, if the ...
1 vote