The kullback-leibler tag has no wiki summary.
15
votes
4answers
2k views
An adaptation of the Kullback-Leibler distance?
Look at this picture:
If we draw a sample from the red density then some values are expected to be less than 0.25 whereas it is impossible to generate such a sample from the blue distribution. As a ...
5
votes
2answers
932 views
Hypothesis testing and total variation distance vs. Kullback-Leibler divergence
In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...
7
votes
2answers
2k views
Kullback-Leibler divergence - interpretation
I have a question about the Kullback-Leibler divergence.
Can someone explain why the "distance" between the blue density and the "red" density is smaller than the distance between the "green" curve ...
8
votes
3answers
3k views
Questions about KL divergence?
I am comparing two distributions with KL divergence which returns me a non-standardized number that, according to what I read about this measure, is the amount of information that is required to ...
11
votes
1answer
1k views
Kullback–Leibler vs Kolmogorov-Smirnov distance
I can see that there are a lot of formal differences between Kullback–Leibler vs Kolmogorov-Smirnov distance measures.
However, both are used to measure the distance between distributions.
Is there ...
5
votes
1answer
201 views
How do I determine how well a dataset approximates a distribution?
Quite simple, I have some probability distribution p(x), how can I measure whether one empirical density (set of delta masses) is a better approximation than another. I know that KL-divergence is a ...
4
votes
2answers
125 views
Connection between Fisher metric and the relative entropy
Can someone prove the following connection between Fisher information metric and the relative entropy (or KL divergence) in a purely mathematical rigorous way?
$$D( p(\cdot , a+da) \| p(.,a) ) ...
3
votes
1answer
86 views
Orthogonal intersection of linear family and exponential family
I asked the following question in MSE for which I couldn't get any answer yet. I thought this would be a better place for that question.
In statistical maniolds ...
1
vote
1answer
75 views
How does one express the decrease in minimal type II error bound for each observation added?
Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions:
$P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian ...
