7
$\begingroup$

Background

I am taking an introductory course on probability and inference. We recently covered several useful inequalities which I will list below:

Markov's Inequality

Let $X$ be a non-negative random variable and suppose $E(X)$ exists. For any $t>0$, $$P(X>t) \leq \frac{E(X)}{t}$$

Extension of Markov's Inequality

Let $Z \sim N(0,1)$. Let $t>0$. Show that, for any $k>0$, $$ P(|Z|>t) \leq \frac{E|Z|^k}{t^k} $$ and, to find the best bound, you can minimize over $k>0$.

Chebyshev's Inequality for $Z\sim N(0,1)$

Let $k > 0$. $$P(|Z|\geq k) \leq \frac{1}{k^2}$$

Mill's Inequality

Let $Z\sim N(0,1)$. Then, $$P(|Z|>t) \leq \sqrt{\frac{2}{\pi}} \frac{e^{-t^2/2}}{t} $$

Question

I ended up plotting several of these bounds out of curiousity to see how they compare when applied to $Z \sim N(0,1)$. I also plotted the true tail probability $P(|Z|>t)$.

enter image description here

I noticed that all of these bounds are practically useless for small $t$ (e.g, $t<0.7$). All the bounds say that $P(|Z|>t) < 1$, which we already knew!

Is there any obvious answer why these common bounds are only useful for larger $t$? Is it harder to get a bound for when $t$ is small?

$\endgroup$
4
$\begingroup$

The first thing to note here is that the upper bounds on the probabilities are going above one in some cases, and obviously you can truncate these to make them one in that Chebychev bound and the Mills bound go above one is really just a matter of convention --- i.e., the expression does not bother to differentiate between the useless bound at unity, and useless bounds above unity.

Setting aside that complication, one way to look at this is to try to find the nastiest distribution in each case --- i.e., the one that achieves the stated bound. For example, suppose we just consider the Chebychev inequality, which applies to any distribution. The uselessness of the bound in this case for small $t$ is the fault of the nefarious Bernoulli distribution! For simplicity, consider the shifted version of this distribution, with probability mass values:

$$\mathbb{P}(Z=-1) = \mathbb{P}(Z=1) = \frac{1}{2}.$$

For this distribution we have:

$$\mathbb{P}(|Z| > t) = \begin{cases} 1 & & \text{if } t < 1, \\[6pt] 0 & & \text{if } t \geqslant 1. \\[6pt] \end{cases}$$

The Chebychev inequality must accommodate this distribution, so it cannot say anything useful at all in the case where $t<1$. In this case, all that can be said is that the tail probability is no greater than one! You can proceed likewise for the other inequalities, trying to find a distribution that acheives the stated bound (or a sequence of distributions that approach it).

$\endgroup$
2
  • $\begingroup$ If I understand you correctly, you're saying that Chebyshev bound has to apply in the same way for every random variable that has mean $\mu$ and standard deviation $\sigma$. Since both $Z\sim N(0,1)$ and the shifted Bernoulli r.v. both have $\mu=0$ and $\sigma=1$, the bound on $P(|X|>t)$ must be greater than 1 for $t<1$ (based on the Bernoulli tail calculation you gave). That's cool! It totally makes sense. It also seems to suggest that we could get a better bounding curve for $P(|Z|>t)$, $Z\sim N(0,1)$ if we specialized the analysis to $N(0,1)$ from the beginning. $\endgroup$ – nwsteg Dec 5 '20 at 0:15
  • 2
    $\begingroup$ If you know that $Z \sim \text{N}(0,1)$ (or indeed, if you have narrowed things down to any single distribution for $Z$) then there is really no need for a bound at all, since you can just compute the probabilities from the distribution. In this case the distribution probabilities are their own "bounds". $\endgroup$ – Ben Dec 5 '20 at 2:10
1
$\begingroup$

The distribution of $\bar X$ near the true mean will depend on the distribution of $X$ near the true mean, so you won't be able to bound it just by making assumptions about moments or other expectations. Also, there's more theoretical interest in tail bounds, so more work has gone into refining them.

Results for $\bar X$ near the mean tend not to be explicit bounds. Edgeworth and Cornish-Fisher expansions are examples: valid for values near the mean, but error only up to some order in $n$. For example, Johnson gives an approximation for confidence intervals on $\bar X$ based on Cornish-Fisher expansions that works well for quite a wide range of distributions, but doesn't give an explicit bound. Saddlepoint expansions are another example: very accurate, but error only bounded up to an unknown constant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.