Simple mathematical relationships like $V(X) = E(X^2) - E(X)^2$, aside from being theoretical results, are useful because they allow analysts to do back-of-the-envelope calculations, restate results for simpler interpretation and visualization, and come up with new statistics on the fly.

I'm asking for a useful expansion or identity, and an example of its usefulness. Obviously there are many; keep it to one identity per answer so that they can be voted on individually.

The identities don't have to be as "simple" as the variance expansion above, but they should be accessible to applied researchers who perhaps don't have training in formal mathematical stats.

They also don't need to be popular or standard; part of the point of this question is to ferret out some more obscure but useful tricks that one might not encounter in an intro stats or regression class.

This is related to What theories should every statistician know? but with the scope constrained to specific mathematical expressions.

  • 2
    $\begingroup$ There is a close vote on this thread as "primarily opinion-based", which is arguably correct. It seems to me that this question could usefully stay open, but maybe it should be made community wiki as a compromise measure. $\endgroup$ – gung - Reinstate Monica Feb 4 '15 at 19:32
  • $\begingroup$ @gung I was meaning for it to be CW all along, but couldn't find a button to do it myself $\endgroup$ – shadowtalker Feb 4 '15 at 19:38
  • $\begingroup$ I just flagged it; the mods can decide. $\endgroup$ – gung - Reinstate Monica Feb 4 '15 at 19:40
  • 1
    $\begingroup$ I'm voting to close this question as off-topic because questions that ask for lists of things do not fit the SE model. Yes, we have allowed certain lists--jokes etc.--even though they should be closed too. But this one strikes me as being particularly egregious in that it has no apparent unifying purpose. $\endgroup$ – whuber Feb 4 '15 at 19:58

You can fill up textbooks answering this question so I'm going to go ahead and dedicate my answer to a few inequalities.

Markov's Inequality: if $X$ is any nonnegative integrable random variable and $a > 0$, then

$$\mathbb{P}(X \geq a) \leq \frac{\mathbb{E}(X)}{a}$$

Chebyshev's Inequality: let $X$ be an integrable random variable with finite expected value $\mu$ and finite non-zero variance $\sigma^2$. Then for any real number $k > 0$ then

$$\mathbb{P}(|X-\mu|\leq k\sigma) \geq \frac{1}{k^2}$$

These two inequalities are very powerful as they are true for any distribution that random variable $X$ comes from. For example, if we know that a random variable has mean 0 and standard deviation 1 we know that the probability that this random variable from an unknown distribution has a value between -2 and 2 must be more than .75.

Cramer-Rao Bound: suppose $\theta$ is an unknown deterministic parameter which is to be estimated from measurements $x$. The variance of any unbiased estimator $\hat{\theta}$ of $\theta$ is then bounded by the reciprocal of the Fisher information $I(\theta)$.


This is powerful because if your unbiased estimated reaches the lower bound then you know your unbiased estimator is the minimum variance unbiased estimator!

Jensen's Inequality: if $X$ is a random variable and $f$ is a convex function, then

$$f\left(\mathbb{E}[X]\right) \leq \mathbb{E}\left[f(X)\right]$$

Like Chebyshev's and Markov's this inequality is applicable all over the place and that's why it's useful!


One such example is a theorem known as the delta method for finding the distribution of a random variable.

The statement is:

Let $\{X_i\}_{i=1}^n$ be a sequence of random variables, and let $$ \sqrt{n}\left(X_i - \mu\right) \overset{d}\rightarrow \operatorname{N} \left(0, \sigma^2\right), \ \forall i. $$ Then, for any continuous function $g$ such that $g'(\mu)$ exists and $g'(\mu) \neq 0$, $$ \sqrt{n}\left(g(X_i) - g(\mu)\right) \overset{d}\rightarrow \operatorname{N} \left(0, g'(\mu)^2\sigma^2\right), \ \forall i. $$

Practically, this means that if a data set is normally distributed with mean $\mu$ and variance $\sigma^2$, then transforming the data by $g$ yields a data set with mean $g(\mu)$ and variance $g'(\mu)^2\sigma^2$.

One simple example of its use is buried in a book describing a program called MARK, on page B-7 (from the book's Appendix B, which also contains a detailed and very accessible derivation and explanation of the method):

Suppose a researcher harvested $N$ fish and computed the average mass of a fish in the sample, $m$, so that one could estimate the total biomass $\hat B$ of the sample with $N \times m$. If the sample is reasonably large, the sampling distribution of $m$ is approximately Gaussian, by the Central Limit Theorem. Then $\hat B$ also has an approximate Gaussian distribution, with standard deviation $N \times \operatorname{SE}(m)$. Knowing this now allows us to conduct comparative statistical tests of biomass in different samples.

  • 1
    $\begingroup$ CLT + delta-method = great trick. Especially if you are good at taking derivatives. $\endgroup$ – shabbychef Feb 4 '15 at 19:17
  • $\begingroup$ @shabbychef I should mention the CLT more explicitly. In fact if you post a CLT answer here I can link to it $\endgroup$ – shadowtalker Feb 4 '15 at 19:26

$e^x \approx 1+x $ when $x$ is very close to 0.

$\frac{e^x}{1+e^x} \approx e^x$ when $x$ is a large negative number.

These are special cases when relative risks are approximately additive risks (case 1) or when odds ratios are approximately relative risks. In epidemiology, risk factors that interest us may have very small effect sizes (RR = 1.04 for instance) and the prevalence of disease may be very rare (for Wilm's tumor, e.g.). Therefore there is a sense of reckless abandon in the language we use to describe results, the error is committed when statisticians forget their effect size and report results in an inconsistent way.


One of the most bayesic ones - the Bayes theorem:

$$ P(\theta | D) = \frac{P(\theta)P(D|\theta)}{P(D)} \varpropto P(\theta)P(D|\theta)$$

You can find multiple threads on Bayes theorem on CV with the explanation, explanations for "lay persons", helpful books, and multiple examples.


$\sum_{i=1}^\infty a^{i-1}$ = $1\over{1-a}$, for (|a| < 1)

This is a useful identity for a number of reasons. One place it can be used is in showing that the geometric distribution is in fact a PMF as follows:

$P(X=x|p)$ = $p(1-p)^{x-1}$, $x=1,2,. . . ,$

To show $\sum_{x=1}^\infty P(X=x)=1$, we can employ the identity:

$\sum_{x=1}^\infty P(X=x)$ = $p\sum_{x=1}^\infty (1-p)^{x-1}$ = $p$$1\over{1-(1-p)}$ = $p\over{p}$ = $1$.

  • $\begingroup$ Thanks for adding several answers. I was asking for examples as well as statements, or at least an explanation if why the statement is useful. $\endgroup$ – shadowtalker Feb 4 '15 at 18:51
  • $\begingroup$ Sorry about that @ssdecontrol. I'm adding comments now. $\endgroup$ – StatsStudent Feb 4 '15 at 19:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.