Upper Bound of MGF for a non-negative random variable with bounded variance

Question

Let $X$ be a non-negative random variable with finite variance. It is obvious that its MGF $E[e^{-\lambda(X-E[X])}]$ exists for $\lambda > 0$.

How to prove that $E[e^{-\lambda(X-E[X])}] \le \exp(\lambda^2 E[X^2]/2)$ for $\lambda > 0$?

Answer 1

The inequality $e^{-u} \leq 1 - u + u^2 /2$ holds for any $u \geq 0$. Replacing $u$ by $\lambda X$ and taking the expectation we get $$ \mathbb{E}[e^{-\lambda X}] \leqslant 1 - \lambda \,\mathbb{E}(X) + \lambda^2\,\mathbb{E}(X^2) /2. $$ Now since $1 + v \leq e^v$ for any $v$, by choosing $v := - \lambda \,\mathbb{E}(X) + \lambda^2 \mathbb{E}(X^2)/2$ $$ \mathbb{E}[e^{-\lambda X}] \leq \exp\left \{-\lambda \,\mathbb{E}(X) + \lambda^2 \,\mathbb{E}(X^2)/2 \right\} $$ which is the result in OP.

Answer 2

Here is a proof adapted from John Duchi's lecture notes, but it's off by a factor of $2$ if you don't mind : We will assume throughout that $X$ has an MGF defined on a neighborhood of $0$, such that all of its moments exist (as finite variance is not sufficient).

Let $X'$ be an independent copy of $X$, i.e. $X'$ has the same distribution as $X$ while being independent of $X$. We have $$E[e^{-\lambda(X-E[X])}] = E_X[e^{-\lambda(X-E_{X'}[X'])}] \le E_XE_{X'}[e^{-\lambda(X-X')}] $$ Where we have applied Jensen's inequality to $e^{\lambda E_{X'}[X']}:= f(E_{X'}[X']) $.

Now for convenience let $Y:= X-X'$. The trick is to notice that $Y$ and $-Y$ are identically distributed, which implies that $Y$ and $S\cdot Y$ are identically distributed when $S$ is an independent Rademacher random variable, i.e. when $S$ only takes value $+1$ and $-1$ with equal probability.

This implies that $$\begin{align*} E_Y[e^{-\lambda Y}] &= E_YE_S[e^{-\lambda S\cdot Y}] \\ &= E_Y[E_S[e^{-\lambda S\cdot Y}\mid Y]]\\ &= E_Y[\cosh(-\lambda Y)] =E_Y[\cosh(\lambda Y)]\end{align*}$$

Now assume that $Y$ is supported on some interval $[-a,a]$. By Taylor's theorem, we know that for all $Y(\omega)$, there exists $\xi\in (0,Y)\cup (Y,0)$ such that $$\cosh(\lambda Y) = 1 + \frac{\lambda^2Y^2}{2} + \frac{\lambda^3Y^3}{6}\sinh(\lambda\xi(Y)) \le 1 + \frac{\lambda^2Y^2}{2} + \frac{\lambda^3Y^3}{6}\sinh(\lambda a)$$

Because $Y$ is symmetric, we have that $E[Y^3]=0$, hence taking expectation we find $$E_Y[\cosh(\lambda Y)] \le 1 + \frac{\lambda^2}{2}E_Y[Y^2]\le \exp\left(\frac{\lambda^2}{2}E_Y[Y^2]\right) $$ Where we have used the well known $1 + x \leq e^{x} $. All that is left is to observe that $E_Y[Y^2] = E_Y[X^2 + (X')^2 - 2XX'] = 2E[X^2] - 2E[X]^2 $ and it follows that $$E[e^{-\lambda(X-E[X])}] \le \exp\left(\lambda^2 (E[X])^2\right)$$

Which is the desired result up to a factor $2$.

---

Now in the general case where $Y$ is not compactly supported, you can construct a sequence $$Y_n = (-n)\vee(Y\wedge n) \in [-n,n]$$ which is bounded and converges pointwise to $Y$. It then follows from Fatou's lemma that $$\begin{align*} E_Y[\cosh(\lambda Y)]&\le \lim\inf_n E_{Y_n}[\cosh(\lambda Y_n)] \\ &\le \lim\inf_n \exp\left(\frac{\lambda^2}{2}E_{Y_n}[Y_n^2]\right)\\ &\le\exp\left(\frac{\lambda^2}{2}E_Y[Y^2]\right)\end{align*}.$$ From which the same conclusion follows.