The kullback-leibler tag has no wiki summary.
0
votes
0answers
22 views
Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?
I need some metric of divergence of two distributions.
(They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right ...
2
votes
1answer
82 views
Is the square root of the symmetric Kullback-Leibler divergence a metric?
It is well known that the square root of the Jensen-Shannon divergence is a true metric, but how about the square root of symmetric KL: D(P||Q)+D(Q||P)? I have reasons to believe that it also is a ...
1
vote
0answers
13 views
Is there any meta-approach for variable selection based of measures of similarity between each two variables?
Is there any meta-approach ( or mayby I should say universal approach which works with different measures ) for variable selection which is based on similarity matrix which every entry ...
2
votes
0answers
29 views
Comparing divergence from uniform distributions with differing supports (discrete)
Imagine we have a potentially biased coin and a potentially biased six-sided die and we want to know which is more biased than the other.
Firstly, is this a reasonable goal? Could it make sense to ...
0
votes
0answers
16 views
Comparison of second-order probability matrices
I've searched over the site for matrix comparison questions, and it seems that the biggest response is 'what do you need it for'.
I have two second-order probability matrices, in which the rows and ...
0
votes
1answer
229 views
Bhattacharyya distance for histograms
One of the ways to measure the similarity of two discrete probability distributions is the Bhattacharyya distance.
In computer vision, for example, it is used to evaluate the degree of similarity ...
4
votes
0answers
54 views
Orthogonal intersection in a Riemannian manifold
Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where ...
3
votes
1answer
86 views
Orthogonal intersection of linear family and exponential family
I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question.
In statistical maniolds ...
4
votes
2answers
124 views
Connection between Fisher metric and the relative entropy
Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way?
$$D( p(\cdot , a+da) \| p(.,a) ) ...
0
votes
1answer
49 views
Calculating Hellinger Divergence from Results of Kernel Density Estimates in Matlab
Using the ksdensity function in matlab returns a density estimation in the form of 2 vectors f and xi. Where f are the density values and xi the corresponding points for the density values.
How do I ...
2
votes
0answers
100 views
Kullback-Leibler vs Hellinger Distance
I am working on this problem in which I have a dataset of n-dimensional examples that come from different and unknown distributions. Given a new sample, I wish to find k examples from the dataset that ...
3
votes
0answers
75 views
Estimating parameters using Kullback-Leibler and Nelder Mead
I want to find the parameters of a model which specifies a set of classification probabilities, for say M classes. (I'll use the parameters in another model later.)
Given a set of parameters ...
0
votes
0answers
152 views
What's the correct way to calculate information gain ratio?
I'm trying to implement information gain ratio[1] to find how much a variable
affects contributes to class membership in a naive bayesian classifier.
I hope to use this for both weighting and to find ...
0
votes
0answers
57 views
Kullback-Leibler Divergence in laymans language
Can some one please explain me in lay mans term what is KL divergence method for divergence and what advantage it has.
Kindly help.
Thank you
0
votes
0answers
104 views
Logistic regression, loss function and KL divergence
In decision theory, a loss function signature is supposed to be
output space * output space -> error
There seems to be many different definition of 'the ...
1
vote
0answers
72 views
On the uniform convergence of relative frequencies of events to their probabilities
I have read the article by Vapnik, Chervonenkis "On the uniform convergence of relative frequencies of events to their probabilities" Theory of Probability and Its Applications, vol XVI, n. , 1971.
...
0
votes
0answers
140 views
Kullback-Leibler divergence for multivariate binomial distributions
I understand KL divergence abstractly, but I'm not exactly sure how you would calculate it for a multivariate binomial distribution (such as an Ising model on a random graph). If I am sampling 100 ...
3
votes
1answer
284 views
Kullback-Leibler divergence: negative values?
Wikipedia - KL properties says that KL can never be negative. But e.g. for texts where the probabilities are very small I somehow get negative values? E.g.
Collection A:
- word count: 321 doc ...
2
votes
0answers
151 views
KL divergence between 2 distributions with unequal cardinalities?
Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.
Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or ...
2
votes
0answers
96 views
Sample distribution for Kullback-Leibler distance
For two $n$ dimensional multivariate normal distributions $X_{1}\sim N\left(\mu_{1},\Sigma_{1}\right)$
and $X_{2}\sim N\left(\mu_{2},\Sigma_{2}\right)$, the Kullback-Leibler distance is given by ...
3
votes
0answers
38 views
Lomax distributions - Kullback Leibler divergence
Does anyone know of a reference for an expression for the Kullback-Leibler divergence between two Lomax (Pareto II) distributions? Not really worried which way the Lomax is parameterized.
0
votes
0answers
150 views
Comparing symmetric KL divergence values
I am trying to understand how to compare KL divergence values. I have 3 sets of data, Good (G), Bad (B) and Unknown (U). I compute the symmetric form of KL for:
Sym_GU = KL(G||B) + KL(B||G)
Sym_BU ...
1
vote
1answer
177 views
Multivariate time series model evaluation with conditional moments
Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula ...
4
votes
2answers
173 views
Maximum entropy sampler
I want to sample from a distribution which has fixed to a given values mean(=0), standard deviation(=1), skewness(=0) and kurtosis. I also want this distribution to be as general as possible, i.e. to ...
2
votes
1answer
213 views
Markov chain convergence, total variation and KL divergence
I have a few related questions regarding the convergence of continuous-state Markov chains.
The theorems that I found claim that Markov chains converge in total variation if they are ...
2
votes
0answers
134 views
Cramer-Rao type bound for Information Gain
I am interested in the Bayes risk of some distribution $\pi$
$$
r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]],
$$
where $L$ is some loss function and $\hat x$ is the ...
3
votes
1answer
540 views
Kullback-Leibler divergence
Suppose we seek to approximate an arbitrary distribution $p_1(x)$ by a normal
$p_2(x) \sim \mathcal N(\mu, \Sigma)$. How can I show that the values that lead to the smallest Kullback–Leibler
...
2
votes
1answer
149 views
Property of KL-divergence
Let $p_1$ and $p_2$ be two distinct probability distributions. Define
$$
L(q)=D(q||p_1)-D(q||p_2)
$$
where $D$ is the usual Kullback-Leibler divergence. Assume the support of $p_2$ is included in ...
1
vote
1answer
286 views
Kullback–Leibler divergence between two wishart distributions
The result is shown in:
[1] W.D. Penny, KL-Divergences of Normal, Gamma, Dirichlet, and Wishart densities, Available at: www.fil.ion.ucl.ac.uk/~wpenny/publications/densities.ps
But could anyone help ...
1
vote
1answer
74 views
How does one express the decrease in minimal type II error bound for each observation added?
Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions:
$P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian ...
5
votes
2answers
931 views
Hypothesis testing and total variation distance vs. Kullback-Leibler divergence
In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
0
votes
0answers
115 views
Shall I use sqrt of J-S divergence or K-L divergence as a measure of deviation from uniform pdf?
Some colleagues are using 'normalized' K-L divergence to measure deviation of unit-area histogram (pseudo discrete pdf) from corresponding uniform distribution ($N$ equal-length bins). Maybe it is ...
2
votes
2answers
229 views
Similarity / dissimilarity of two large bimodal datasets
I am interested in assessing the divergence, or similarity or dissimilarity of 2 datasets that are the results of 2 different lidar instrument measurements. Each dataset has over 90,000 values and ...
5
votes
3answers
2k views
Measure of similarity or distance between two symmetric covariance matrices
are there any measures of similarity or distance between two symmetric covariance matrices (both having the same dimension)?
I am thinking here of analogues to KL divergence of two probability ...
4
votes
3answers
962 views
How to compute the Kullback-Leibler divergence when the PMF contains 0s?
I have the following timeseries
obtained using the data posted below.
For a sliding window size of 10, I am trying to compute the KL-divergence between the PMF of values within the current sliding ...
5
votes
1answer
201 views
How do I determine how well a dataset approximates a distribution?
Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
2
votes
0answers
122 views
How to to calculate the topic distribution of a document
I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document.
I do not know how to calculate ...
7
votes
3answers
678 views
Kullback–Leibler divergence between two gamma distributions
Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf
$g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$
The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and ...
11
votes
1answer
1k views
Kullback–Leibler vs Kolmogorov-Smirnov distance
I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures.
However, both are used to measure the distance between distributions.
Is there ...
2
votes
2answers
3k views
KL divergence between two univariate Gaussians
I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p).
...
15
votes
4answers
2k views
An adaptation of the Kullback-Leibler distance?
Look at this picture:
If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
4
votes
1answer
734 views
Estimate the Kullback-Leibler divergence
I would like to be sure I am able to compute the KL divergence based on a sample.
Assume the data come from a Gamma distribution with shape=1/.85 and scale=.85.
set.seed(937)
theta <- ...
7
votes
2answers
2k views
Kullback-Leibler divergence - interpretation
I have a question about the Kullback-Leibler divergence.
Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
3
votes
1answer
244 views
Properties of Battacharyya distance vs Kullback-Leibler divergence
What properties do these measures have and how can I determine which one is better for a given purpose? What are extreme cases where they differ a lot?
8
votes
3answers
3k views
Questions about KL divergence?
I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
