Questions tagged [divergence]

a function that establishes the "distance" of one probability distribution to the other on a statistical manifold.

Filter by
Sorted by
Tagged with
1
vote
0answers
24 views

What is the range of $a $ such that $\frac{logN}{N^{3a+1/2}}\rightarrow 0$ as $N\rightarrow \infty$? [migrated]

I want to find the range of $a$ such that $\frac{logN}{N^{3a+1/2}}\rightarrow 0$ as $N\rightarrow\infty$. The answer to this question hinges on conditions under which $logN$ diverges slower than the ...
0
votes
0answers
41 views

What are the advantages of Wasserstein distance compared to Jensen-Shannon divergence?

What is the practical difference between Wasserstein metric and Jensen-Shannon divergence? Wasserstein metric is also referred to as Earth mover's distance. From Wikipedia: Wasserstein metric is a ...
1
vote
0answers
10 views

Strong data processing inequality in multiplicative channels

We know that postprocessing will not increase the information. For two random variables $X$ and $Y$, $D(X||Y)>= D(f(X)||f(Y))$ for any operation $f()$ and divergence $D$. A strong data processing ...
2
votes
1answer
55 views

Optimizing forward/reverse KL divergence for Gaussian distributions

The forward/reverse formulations of KL divergence are distinguished by having mean/mode-seeking behavior. The typical example for using KL to optimize a distribution $Q_\theta$ to fit a distribution $...
1
vote
0answers
26 views

Is there a rate of change performance measure for KL-divergence?

In the example figure below, KL-divergence is being used to measure how far the distribution of different parameterizations of Poisson are from an empirical distribution (real data). The minimum of ...
0
votes
0answers
49 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
14 views

Can $f$-divergences narrow the discrepancy between train and test fits in machine learning?

Machine learning models whose task is to predict unseen test data would work best if the test data's distribution turns out to be the same as the training data's distribution. Real data seldom works ...
1
vote
0answers
33 views

Statistical distance between two matrices

The statistical distance between two probability distributions can be measured with $f$-divergences such as the KL-divergence. The statistical distance between two clusters can be measured with ...
1
vote
0answers
29 views

Is KL-divergence just the multiplication rule for independent events, reformulated in terms of entropy?

We know KL-divergence is sometimes expressed like this: which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-...
0
votes
0answers
14 views

Is there any divergence like Jensen-Shannon for two vectors which are not distribution? [duplicate]

I know that the Jensen-Shannon is defined as a divergence between two or more distributions ($P_1,P_2,...P_k$). But, instead of distributions, I have some multiplications of two distributions (so they ...
0
votes
0answers
31 views

What is the correct way to implement Jensen-Shannon Distance?

I'm trying to use this code to compute the Jensen-Shannon distance: ...
0
votes
0answers
18 views

Finding the lower bound for relative entropy $D(f||g)$ where, $f$, $g$ are two different distribution?

I am trying to find a tighter bound of the relative entropy $D(f||g)$. Problem statement: Let, $f$ and $g$ be two discrete probability distribution. $f \in \left[ {Q(x + a),Q( - x - a)} \right]$ and $...
2
votes
1answer
41 views

Sensitivity of KL Divergence

I am very new to the concept of KL divergence. Although I have grasped the fundamental formulations, I have a confusion comparing the KL divergence across the different distributions. Suppose I have 3 ...
0
votes
0answers
25 views

Does minimizing KL-divergence result in maximum entropy principle?

The Kullback-Leibler divergence (or relative entropy) is a measure of how a probability distribution differs from another reference probability distribution. I want to know what connection it has to ...
2
votes
0answers
22 views

Is limiting density of discrete points (LDDP) equivalent to negative KL-divergence?

Is limiting density of discrete points (LDDP), which is a corrected version of differential entropy, equivalent to the negative KL-divergence (or relative entropy) between a density function $m(x)$ ...
3
votes
1answer
48 views

Is relative entropy equal to cross-entropy during optimization?

I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy ...
0
votes
0answers
13 views

Power-density divergence for logistic regression

How is power-density (or $\beta$-divergence) written for the binary logistic regression problem?
2
votes
0answers
23 views

Show that a sequence of random variables diverges to infinity in probability

I have sequences of real-valued random variables $\{X_T\}, \{Y_T\}$ and a sequence of real numbers $\{a_T\}$. As $T\rightarrow\infty$, I know that $$ a_T \rightarrow \infty $$ and $$ X_T \overset{d}{\...
0
votes
0answers
44 views

Understanding concrete dropout

I am trying to understand a technique call Concrete Dropout for automatically tuning the dropout rate during the network training. However, I am unable to follow the work described in the paper ...
1
vote
0answers
26 views

What are the best known techniques to verify that a GAN samples correctly from a given distribution?

I would like to know what are the best known techniques to check that a generative adversarial network (GAN) samples from the correct distribution. Naively, I would say it all boils down to a ...
0
votes
1answer
21 views

Lemma KL-Divergence (Differential Privacy)

I am studying differential privacy and I got stuck again in proof of a lemma. Which is: "$D_{\infty}^\delta(Y||Z) \leq \epsilon$ if and only if there exists a random variable $Y'$ such that $\...
0
votes
0answers
12 views

Divergence of a model and its variables

I am reading a model documentation which gives the "Divergence" of the model and all its parameters. For example, Model Divergence = 1.529 Divergence of Variable1 = 0.015 Divergence of ...
1
vote
0answers
16 views

Are there differentiable estimators for Entropy?

I have recently came across a paper on estimation of Information theoretic measure such as Entropy, Mutual Information and divergence, using a Mean Nearest Neighbor approach. Since, the estimator is ...