All Questions

Filter by
Sorted by
Tagged with
2
votes
0answers
498 views

Upper bound on KL divergence

Is there a maximum (unique?) to the KL divergence between discrete distributions p & q, with the restriction that q is a proper probability distribution? I know KL is unbounded from above when q ...
4
votes
1answer
1k views

KL divergence bounds square of L1 norm

In Cover & Thomas, Elements of Information Theory, at the section on Conditional Limit Theorem (11.6), it is proved that the KL divergence bounds the $\cal{L}_1$-norm from above, $\frac{1}{2\ln2}\...
5
votes
0answers
203 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler distance, so we have: $$ KL(p,q) = \sum_{s \in S} p(s) \log\left( \...
2
votes
0answers
265 views

Kullback Leibler divergence “efficient” upper bound

For a distribution of N values, how can I efficiently upper-bound the largest divergence between all non-negative distributions over the same random field? For example, for all distributions of a ...
1
vote
1answer
171 views

How does one express the decrease in minimal type II error bound for each observation added?

Problem: I have a "classifier" that uses some arbitrary hypothesis test on observations from one of two known probability distributions: $P_0$ (null hypothesis $H_0$) is a zero-mean Gaussian $\...
10
votes
2answers
5k views

Hypothesis testing and total variation distance vs. Kullback-Leibler divergence

In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples ...