5
$\begingroup$

Let $X$ be a non-negative random variable with finite variance. It is obvious that its MGF $E[e^{-\lambda(X-E[X])}]$ exists for $\lambda > 0$.

How to prove that $E[e^{-\lambda(X-E[X])}] \le \exp(\lambda^2 E[X^2]/2)$ for $\lambda > 0$?

$\endgroup$
1
  • 1
    $\begingroup$ $E[e^{-\lambda(X - E[X])}]$ is not MGF: $E[e^{\lambda X}]$ is. $\endgroup$
    – Zhanxiong
    Commented May 17, 2023 at 20:53

2 Answers 2

5
$\begingroup$

The inequality $e^{-u} \leq 1 - u + u^2 /2$ holds for any $u \geq 0$. Replacing $u$ by $\lambda X$ and taking the expectation we get $$ \mathbb{E}[e^{-\lambda X}] \leqslant 1 - \lambda \,\mathbb{E}(X) + \lambda^2\,\mathbb{E}(X^2) /2. $$ Now since $1 + v \leq e^v$ for any $v$, by choosing $v := - \lambda \,\mathbb{E}(X) + \lambda^2 \mathbb{E}(X^2)/2$ $$ \mathbb{E}[e^{-\lambda X}] \leq \exp\left \{-\lambda \,\mathbb{E}(X) + \lambda^2 \,\mathbb{E}(X^2)/2 \right\} $$ which is the result in OP.

$\endgroup$
2
  • 1
    $\begingroup$ Great answer, which fixes my previous answer's hole nicely! (+1) $\endgroup$
    – Zhanxiong
    Commented May 18, 2023 at 13:20
  • $\begingroup$ Short and sweet, very nice :) (+1) $\endgroup$ Commented May 18, 2023 at 14:15
1
$\begingroup$

Here is a proof adapted from John Duchi's lecture notes, but it's off by a factor of $2$ if you don't mind : We will assume throughout that $X$ has an MGF defined on a neighborhood of $0$, such that all of its moments exist (as finite variance is not sufficient).

Let $X'$ be an independent copy of $X$, i.e. $X'$ has the same distribution as $X$ while being independent of $X$. We have $$E[e^{-\lambda(X-E[X])}] = E_X[e^{-\lambda(X-E_{X'}[X'])}] \le E_XE_{X'}[e^{-\lambda(X-X')}] $$ Where we have applied Jensen's inequality to $e^{\lambda E_{X'}[X']}:= f(E_{X'}[X']) $.

Now for convenience let $Y:= X-X'$. The trick is to notice that $Y$ and $-Y$ are identically distributed, which implies that $Y$ and $S\cdot Y$ are identically distributed when $S$ is an independent Rademacher random variable, i.e. when $S$ only takes value $+1$ and $-1$ with equal probability.

This implies that $$\begin{align*} E_Y[e^{-\lambda Y}] &= E_YE_S[e^{-\lambda S\cdot Y}] \\ &= E_Y[E_S[e^{-\lambda S\cdot Y}\mid Y]]\\ &= E_Y[\cosh(-\lambda Y)] =E_Y[\cosh(\lambda Y)]\end{align*}$$

Now assume that $Y$ is supported on some interval $[-a,a]$. By Taylor's theorem, we know that for all $Y(\omega)$, there exists $\xi\in (0,Y)\cup (Y,0)$ such that $$\cosh(\lambda Y) = 1 + \frac{\lambda^2Y^2}{2} + \frac{\lambda^3Y^3}{6}\sinh(\lambda\xi(Y)) \le 1 + \frac{\lambda^2Y^2}{2} + \frac{\lambda^3Y^3}{6}\sinh(\lambda a)$$

Because $Y$ is symmetric, we have that $E[Y^3]=0$, hence taking expectation we find $$E_Y[\cosh(\lambda Y)] \le 1 + \frac{\lambda^2}{2}E_Y[Y^2]\le \exp\left(\frac{\lambda^2}{2}E_Y[Y^2]\right) $$ Where we have used the well known $1 + x \leq e^{x} $. All that is left is to observe that $E_Y[Y^2] = E_Y[X^2 + (X')^2 - 2XX'] = 2E[X^2] - 2E[X]^2 $ and it follows that $$E[e^{-\lambda(X-E[X])}] \le \exp\left(\lambda^2 (E[X])^2\right)$$

Which is the desired result up to a factor $2$.


Now in the general case where $Y$ is not compactly supported, you can construct a sequence $$Y_n = (-n)\vee(Y\wedge n) \in [-n,n]$$ which is bounded and converges pointwise to $Y$. It then follows from Fatou's lemma that $$\begin{align*} E_Y[\cosh(\lambda Y)]&\le \lim\inf_n E_{Y_n}[\cosh(\lambda Y_n)] \\ &\le \lim\inf_n \exp\left(\frac{\lambda^2}{2}E_{Y_n}[Y_n^2]\right)\\ &\le\exp\left(\frac{\lambda^2}{2}E_Y[Y^2]\right)\end{align*}.$$ From which the same conclusion follows.

$\endgroup$
5
  • $\begingroup$ Thank you! Your answer is quite helpful! $\endgroup$ Commented Apr 17, 2023 at 13:48
  • $\begingroup$ The inequality $Y^3 \sinh(\lambda \xi) \leq Y^3 \sinh(\lambda a)$ does not seem so easy since $Y$ can be negative and $\xi$ depends on $Y$. Maybe a justification could help? $\endgroup$
    – Yves
    Commented May 18, 2023 at 8:35
  • $\begingroup$ @Yves : that's because I picked $\xi$ between $0$ and $Y$, so $Y^3 \sinh(\lambda \xi) $ is always non-negative, and then the upper bound follows from the fact that $Y\le a$ almost surely and that $\sinh$ is non-decreasing. $\endgroup$ Commented May 18, 2023 at 10:47
  • $\begingroup$ When $Y$ is negative I get $-\sinh(\lambda a) \leq \sinh(\lambda \xi) \leq 0$ because $-a \leq \xi \leq 0$ and $\sinh(-\lambda a) = - \sin(\lambda a)$. But then by multiplying by $Y^3 \leq 0$ I get $0 \leq Y^3 \sinh(\lambda \xi) \leq -Y^3 \sinh(\lambda a)$ which still puzzles me. $\endgroup$
    – Yves
    Commented May 18, 2023 at 12:19
  • $\begingroup$ $-Y^3\sinh(\lambda a) = Y^3\sinh(-\lambda a) \le Y^3\sinh(\lambda a) $ so seems like there is no problem... Or am I missing something ? $\endgroup$ Commented May 18, 2023 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.