All Questions

Filter by
Sorted by
Tagged with
5
votes
1answer
175 views

Why is Kullback-Leilbler divergence a better metric for measuring distance between two probability distributions than squared error? [duplicate]

I know that KL-divergence is a metric that is more suitable when we want to measure the distance between numbers which a probability form. However, I am still confused what is the benefit of using KL-...
2
votes
0answers
63 views

Relation Between Wasserstein Distance and Relative Entropy

Consider the Wasserstein metric of order one $W_1$ (aka the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and relative entropy and what this would mean intuitively. ...
1
vote
0answers
156 views

KL divergence between sample and true (multivariate normal) distribution

I was wondering, whether there is a possible interpretation of the KL-Divergence between sample and true distribution in terms of probabilities. E.g. given $P=\mathcal{N}\left(\mu,\Sigma\right)$ and $...
4
votes
2answers
566 views

Correcting Kullback-Leibler divergence for size of datasets

We have the following implementation of KLD: ...
1
vote
0answers
54 views

Distance between angle distributions

I want to quantify the complexity of the street network of different cities. For each city I have the angle distribution of its streets. My hypothesis that the more complex the street network, the ...
13
votes
3answers
3k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
1
vote
1answer
288 views

Calculate the Kullback-Leibler Divergence for these 2 Gamma distributions

I have 2 models $P \sim Ga(115,1329.914) \\ Q \sim Ga(140,650.6775)$ and I'm looking to calculate the K-L divergence of these 2. $D_{KL}(P||Q) = \int_\infty ^\infty p(x) log \frac{p(x)}{q(x)}\,dx$...
2
votes
0answers
203 views

Distance or divergence for ordinal distribution

Measures like KL divergence can be symmetrized (into JS divergence). Bhattacharyya distance serves a similar function. Either is well-suited to both continuous distributions and discrete (e.g. ...
0
votes
0answers
44 views

Lower bound for difference of probabilities of a same event under two different distributions

Say $P$ and $Q$ are two probabilities distributions on $[n]$. I can upper bound the difference of the probability of an event $A$ under $P$ and $Q$ by the total variation distance between $P$ and $Q$. ...
0
votes
0answers
75 views

Derive a constant in Kullback-Liebler divergence proof

From Kullback-Liebler divergence of matrix factorization; \begin{equation*} \mathrm{X}\approx\mathbf{WH} \tag{1} \end{equation*} How equation $(2)$ is derived to constant equality in equation $(3)$? ...
2
votes
1answer
759 views

KL divergence of multivariate lognormal distributions

I've been trying to get the KL divergence for two lognormal distributions. I know what it is for the univariate case, $$ D(f_i\|f_j)= \frac1{2\sigma_j^2}\left[(\mu_i-\mu_j)^2+\sigma_i^2-\sigma_j^2\...
2
votes
2answers
2k views

Kullback-Leibler distance for comparing two distribution from sample points

I have two data samples of a value and I want to compute some distance which would represent the difference in their distribution. I read about Kullback-Leibler distance which could be used for ...
3
votes
1answer
55 views

How does one quantify the difference between two distributions, especially if sample sizes differ?

I have plotted some experimental data of mine, and these data points fall into the following distributions: So, these are fairly non-trivial looking distributions. I would like to figure out methods ...
3
votes
1answer
604 views

KL Divergence, Bregman, and uniqueness

While reading the following paper on Bregman Divergence (link) Banerjee, Arindam, et al. "Clustering with Bregman divergences." Journal of machine learning research 6.Oct (2005): 1705-1749. In ...
3
votes
1answer
27 views

What is the factor equal to if the true and empirical distribution both are 0 for a configuration?

Suppose I want to calculate the relative entropy: $$D(q||p)= \sum q(x)\log \frac{q(x)}{p(x)}$$ If, for some $x$, I have $q(x)=p(x)=0$, does the corresponding factor in the sum becomes $0$?
0
votes
1answer
231 views

How to apply distance metrics to compare bar plot (nominal histogram) data

I have a data set for libraries, I would like to find the (Similarity / dissimilarity) among it based on book category, so for each category there is single value represent the number of books that ...
46
votes
5answers
13k views

Intuition on the Kullback-Leibler (KL) Divergence

I have learned about the intuition behind the KL Divergence as how much a model distribution function differs from the theoretical/true distribution of the data. The source I am reading goes on to say ...
3
votes
3answers
660 views

Quantify Difference/Distance between Lognormal distributions

I am trying to determine a metric that quantifies the distance between two continuous lognormal distributions. The data is actually a mixture of two lognormal distributions (I am not sure if this can ...
2
votes
0answers
82 views

Occupancy octree metrics (Kullback-Leibler)

As I'm currently working on scan matching for outdoor environments I was wondering about the best metric to compare two occupancy octrees (one resulted from the scan matching and one ground truth ...
14
votes
3answers
11k views

Calculate the Kullback-Leibler Divergence in practice?

I am using KL Divergence as a measure of dissimilarity between 2 $p.m.f.$ $P$ and $Q$. $$D_{KL}(P||Q) = \sum_{i=1}^N \ln \left( \frac{P_i}{Q_i} \right) P_i$$ $$=-\sum P(X_i)ln\left(Q(X_i)\right) + \...
1
vote
2answers
4k views

Bhattacharyya distance for histograms

One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance. In computer vision, for example, it is used to evaluate the degree of similarity ...