3
$\begingroup$

I'm reading the paper, T. van Erven and P. Harremoës, Rényi Divergence and Kullback-Leibler Divergence, arXiv 1206.2459 on the Renyi divergence, and I'm trying to make sense of "Example 1". I think that I'm just getting messed up on the terminology, so I'm not able to derive that the identity is true.

Am I missing some assuptions/definitions/notational conventions so that this doesn't make sense?

Is there a different way of describing/deriving their "Example 1"?

Example 1: Let $Q$ be a probability distribution and $A$ a set with positive probability.
Let $P$ be the conditional distribution of $Q$ given $A$.
Then $$ D_\alpha( P \Vert Q ) = - \ln Q(A). $$

Some features of their notation (from earlier in the paper): $$ P=(p_1, p_2, \dots, p_n) \\ Q=(q_1, q_2, \dots, q_n) $$ refer to (discrete) probability distributions.

The Renyi divergence itself is expressed as: $$ D_\alpha ( P \Vert Q ) = \frac{1}{1-\alpha} \ln \sum_{i=1}^{n} p_i^{\alpha} q_i^{1-\alpha} $$

I don't understand the notion of "the conditional distribution of $Q$ given $A$", to me, this reads as "the probability distribution of a probability distribution [function]".

I'm also unclear on the interpretation the result: the result should be the number, but what's written (looks like it) is the log of a set (note the definition of $A$).

$\endgroup$
2
$\begingroup$

By "the conditional distribution of $Q$ given $A$" they mean: $$ p_i = \frac{q_i}{\sum_{i\in A} q_i}\text{ if } i \in A $$ and zero otherwise. The denominator is just normalization, so that $p_i$ sum up to $1$.

With that defined $P$ and $Q$ the divergence yields $$ \frac{1}{1-\alpha}\ln\sum_{i\in A} \left( \frac{q_i^\alpha}{(\sum_{j\in A} q_j)^{\alpha}}q_i^{1-\alpha} \right) $$ and separating the logarithm with respect to two sums we get $$ \ln \sum_{i\in A}q_i. $$ So, $Q(A)$ is probability of $Q$ being in $A$ (or, equivalently, probability summed over set $A$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.