4
$\begingroup$

$X$ is a measurable mapping from a discrete sample space $\Omega$ to $\mathbb R$.

$\mu_1$ and $\mu_2$ are two probability measures on $\Omega$. Assume they have probability mass functions $f_1$ and $f_2$.

Is $E_{\mu_1} X - E_{\mu_2} X \leq E_{\mu_1} \log (\frac{f_1}{f_2})$ i.e. $KL(\mu_1|\mu_2)$? I am not sure about it, since I can always scale $X$ arbitrarily.

I hope to relate the LHS to KL divergence in some way: $E_{\mu_1} X - E_{\mu_2} X \leq h(KL(\mu_1|\mu_2))$, for some function $h$.

Thanks.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

If $\mu_1=\mu_2$ then both sides equal $0$. This is the only situation in which the inequality is guaranteed to hold.

To see this, pick an $\omega\in\Omega$ for which $\mu_1(\omega) \ne \mu_2(\omega)$, let $x\in \mathbb{R}$, and define

$$X(\omega) = x$$

and $X(\omega)= 0$ otherwise. $X$ is measurable because the space is discrete. Compute

$$\mathbb{E}_{\mu_1}(X) = \mu_1(\{\omega\})x = f_1(\omega)x;\quad \mathbb{E}_{\mu_2}(X) = \mu_2(\{\omega\})x = f_2(\omega)x,$$

whence

$$\mathbb{E}_{\mu_1}(X) - \mathbb{E}_{\mu_2}(X) = \left(f_1(\omega) - f_2(\omega)\right)x.$$

Since the difference in parentheses is nonzero, the right hand side can be made equal to any real number $y$ by choosing

$$x = \frac{y}{f_1(\omega) - f_2(\omega)}.$$

Improve this answer
$\endgroup$
6
  • $\begingroup$ I see the proof for "if-part" but not the "only-if" part. $\endgroup$
    – user1669710
    Commented Sep 23, 2014 at 17:25
  • 1
    $\begingroup$ @user1669710 I don't follow: I haven't made any statement of equivalence. The assertion is that when the measures differ, then the inequality cannot hold for all random variables $X$. $\endgroup$
    – whuber ♦
    Commented Sep 23, 2014 at 17:29
  • $\begingroup$ Thanks; Also, I totally missed the "discrete sample space" part. I may still be incorrect, but I was wondering, in the continuous case if we define two different probability measures such that the expectations are equal, where as the KL-divergence is not. In which case, it looks like the inequality may hold even when the measures are not the same. $\endgroup$
    – user1669710
    Commented Sep 23, 2014 at 18:11
  • 1
    $\begingroup$ @user1669710 The same argument applies to any measures: if they differ, then random variables that assign huge values to elements of sets where the measures differ (and otherwise equal zero) can have arbitrary differences in their expectations. There simply is no useful connection between expectations, which depend on random variables, and the KL-divergence, which does not refer to random variables at all. $\endgroup$
    – whuber ♦
    Commented Sep 23, 2014 at 18:17
  • 1
    $\begingroup$ Because the left hand side can be any real number, what do you think? $\endgroup$
    – whuber ♦
    Commented Sep 23, 2014 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.