17
$\begingroup$

This question arises from the one asked here about a bound on moment generating functions (MGFs).

Suppose $X$ is a bounded zero-mean random variable taking on values in $[-\sigma, \sigma]$ and let $G(t) = E[e^{tX}]$ be its MGF. From a bound used in a proof of Hoeffding's Inequality, we have that $$G(t) = E[e^{tX}] \leq e^{\sigma^2t^2/2}$$ where the right side is recognizable as the MGF of a zero-mean normal random variable with standard deviation $\sigma$. Now, the standard deviation of $X$ can be no larger than $\sigma$, with the maximum value occurring when $X$ is a discrete random variable such that $P\{X = \sigma\} = P\{X = -\sigma\} = \frac{1}{2}$. So, the bound referred to can be thought of as saying that the MGF of a zero-mean bounded random variable $X$ is bounded above by the MGF of a zero-mean normal random variable whose standard deviation equals the maximum possible standard deviation that $X$ can have.

My question is: is this a well-known result of independent interest that is used in places other than in the proof of Hoeffding's Inequality, and if so, is it also known to extend to random variables with nonzero means?

The result that prompts this question allows asymmetric range $[a,b]$ for $X$ with $a < 0 < b$ but does insist on $E[X] = 0$. The bound is $$G(t) \leq e^{t^2(b-a)^2/8} = e^{t^2\sigma_{max}^2/2}$$ where $\sigma_{\max} = (b-a)/2$ is the maximum standard deviation possible for a random variable with values restricted to $[a,b]$, but this maximum is not attained by zero-mean random variables unless $b = -a$.

$\endgroup$
  • 6
    $\begingroup$ Random variables that satisfy bounds on the mgf like the one you quote are called subgaussian random variables. They play a central role, e.g., in nonasymptotic random matrix theory and some associated results in compressed sensing. See, e.g., the link in the answer here. (This obviously doesn't speak to your particular question; but, it is of a related nature.) $\endgroup$ – cardinal Jan 16 '12 at 3:59
  • $\begingroup$ @cardinal, that's interesting because I sometimes see bounds that apply only to bounded random variables, and they're useful enough that people will prove things by splitting a random variable into bounded and unbounded parts; and you're telling me that all bounded random variables are subgaussian, so I wonder how many of the inequalities I've seen for bounded random variables have generalizations to subgaussian random variables. $\endgroup$ – user54038 Feb 21 '20 at 0:37
5
$\begingroup$

I can't answer the first part of your question, but as for extending it to random variables with nonzero means...

First, note that any r.v. $Z$ with finite range $[a+\mu,b+\mu]$ and (necessarily finite) mean $\mu$ can be transformed into an r.v. $X = Z-\mu$ that is, of course, zero mean with range $[a,b]$ (thus satisfying the conditions in your problem statement). The transformed variate has m.g.f. $\phi_X(t) = \exp\{-\mu t\}\phi_Z(t)$ (by basic properties of the m.g.f.) Multiplying both sides by $\exp\{\mu t\}$ and applying the inequality gives:

$\phi_Z(t) = \exp\{\mu t\}\phi_X(t) \leq \exp\{\mu t\}\exp\{t^2\sigma^2_\text{max}/2\} = \exp\{\mu t + t^2\sigma^2_\text{max}/2\} $

Not surprisingly, the m.g.f. of a Normal random variable with the same mean and standard deviation equal to $\sigma_\text{max}$.

$\endgroup$
3
$\begingroup$

Since $e^{ t x}$ is a convex function, by Jensen's inequality, we have $$e^{t X} \le \frac{\sigma+X}{2\sigma}e^{-t\sigma}+ \frac{\sigma-X}{2\sigma}e^{t\sigma}$$

Taking expectation of both sides of above inequality we get $$E[ e^{t X}] \le \frac{1}{2}e^{-t\sigma} + \frac{1}{2}e^{t\sigma}$$ where we used the zero-mean assumption on $X$. It remains to prove that $\frac{1}{2}(e^y+e^{-y})\le e^{y^2/2}$. (Then, replacing $y=t\sigma$ we arrive at $E[ e^{t X}] \le e^{t^2\sigma^2/2}$.)

Here is my proof for $$\quad \quad e^y+e^{-y}\le 2 e^{y^2/2} \quad \quad \text{for }y \in \mathbb{R} \quad (1)$$.

By Taylor expansion of $e^y$ we have $$e^y + e^{-y} = \sum_{n=0}^{\infty} \frac{y^n}{n!} + \sum_{n=0}^{\infty} \frac{(-1)^ny^n}{n!} =2\sum_{k=0}^{\infty} \frac{y^{2k}}{(2k)!} $$ $$2e^{y^2/2} = 2\sum_{k=0}^{\infty} \frac{y^{2k}}{2^k k!}$$ from which we get (1) since $(2k)! \ge 2^k k!$ (this can be shown by induction or Stirling bounds on k!).

س

$\endgroup$
  • $\begingroup$ Welcome to CV. When you delimit $\TeX$ markup with dollar signs $ it will be rendered nicely. I inserted them for you. It's unclear how your first inequality is an application of Jensen's inequality, though: would you mind showing it more explicitly? $\endgroup$ – whuber Feb 20 '20 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.