Which expansions and identities are useful to applied statisticians? [closed]

Question

Simple mathematical relationships like $V(X) = E(X^2) - E(X)^2$, aside from being theoretical results, are useful because they allow analysts to do back-of-the-envelope calculations, restate results for simpler interpretation and visualization, and come up with new statistics on the fly.

I'm asking for a useful expansion or identity, and an example of its usefulness. Obviously there are many; keep it to one identity per answer so that they can be voted on individually.

The identities don't have to be as "simple" as the variance expansion above, but they should be accessible to applied researchers who perhaps don't have training in formal mathematical stats.

They also don't need to be popular or standard; part of the point of this question is to ferret out some more obscure but useful tricks that one might not encounter in an intro stats or regression class.

This is related to What theories should every statistician know? but with the scope constrained to specific mathematical expressions.

Answer 1

You can fill up textbooks answering this question so I'm going to go ahead and dedicate my answer to a few inequalities.

Markov's Inequality: if $X$ is any nonnegative integrable random variable and $a > 0$, then

$$\mathbb{P}(X \geq a) \leq \frac{\mathbb{E}(X)}{a}$$

Chebyshev's Inequality: let $X$ be an integrable random variable with finite expected value $\mu$ and finite non-zero variance $\sigma^2$. Then for any real number $k > 0$ then

$$\mathbb{P}(|X-\mu|\leq k\sigma) \geq \frac{1}{k^2}$$

These two inequalities are very powerful as they are true for any distribution that random variable $X$ comes from. For example, if we know that a random variable has mean 0 and standard deviation 1 we know that the probability that this random variable from an unknown distribution has a value between -2 and 2 must be more than .75.

Cramer-Rao Bound: suppose $\theta$ is an unknown deterministic parameter which is to be estimated from measurements $x$. The variance of any unbiased estimator $\hat{\theta}$ of $\theta$ is then bounded by the reciprocal of the Fisher information $I(\theta)$.

$$\mathrm{var}(\hat{\theta})\geq\frac{1}{I(\theta)}$$

This is powerful because if your unbiased estimated reaches the lower bound then you know your unbiased estimator is the minimum variance unbiased estimator!

Jensen's Inequality: if $X$ is a random variable and $f$ is a convex function, then

$$f\left(\mathbb{E}[X]\right) \leq \mathbb{E}\left[f(X)\right]$$

Like Chebyshev's and Markov's this inequality is applicable all over the place and that's why it's useful!

Answer 2

One such example is a theorem known as the delta method for finding the distribution of a random variable.

The statement is:

Let $\{X_i\}_{i=1}^n$ be a sequence of random variables, and let $$ \sqrt{n}\left(X_i - \mu\right) \overset{d}\rightarrow \operatorname{N} \left(0, \sigma^2\right), \ \forall i. $$ Then, for any continuous function $g$ such that $g'(\mu)$ exists and $g'(\mu) \neq 0$, $$ \sqrt{n}\left(g(X_i) - g(\mu)\right) \overset{d}\rightarrow \operatorname{N} \left(0, g'(\mu)^2\sigma^2\right), \ \forall i. $$

Practically, this means that if a data set is normally distributed with mean $\mu$ and variance $\sigma^2$, then transforming the data by $g$ yields a data set with mean $g(\mu)$ and variance $g'(\mu)^2\sigma^2$.

One simple example of its use is buried in a book describing a program called MARK, on page B-7 (from the book's Appendix B, which also contains a detailed and very accessible derivation and explanation of the method):

Suppose a researcher harvested $N$ fish and computed the average mass of a fish in the sample, $m$, so that one could estimate the total biomass $\hat B$ of the sample with $N \times m$. If the sample is reasonably large, the sampling distribution of $m$ is approximately Gaussian, by the Central Limit Theorem. Then $\hat B$ also has an approximate Gaussian distribution, with standard deviation $N \times \operatorname{SE}(m)$. Knowing this now allows us to conduct comparative statistical tests of biomass in different samples.

Answer 3

$e^x \approx 1+x $ when $x$ is very close to 0.

$\frac{e^x}{1+e^x} \approx e^x$ when $x$ is a large negative number.

These are special cases when relative risks are approximately additive risks (case 1) or when odds ratios are approximately relative risks. In epidemiology, risk factors that interest us may have very small effect sizes (RR = 1.04 for instance) and the prevalence of disease may be very rare (for Wilm's tumor, e.g.). Therefore there is a sense of reckless abandon in the language we use to describe results, the error is committed when statisticians forget their effect size and report results in an inconsistent way.

Answer 4

One of the most bayesic ones - the Bayes theorem:

$$ P(\theta | D) = \frac{P(\theta)P(D|\theta)}{P(D)} \varpropto P(\theta)P(D|\theta)$$

You can find multiple threads on Bayes theorem on CV with the explanation, explanations for "lay persons", helpful books, and multiple examples.

Answer 5

$\sum_{i=1}^\infty a^{i-1}$ = $1\over{1-a}$, for (|a| < 1)

This is a useful identity for a number of reasons. One place it can be used is in showing that the geometric distribution is in fact a PMF as follows:

$P(X=x|p)$ = $p(1-p)^{x-1}$, $x=1,2,. . . ,$

To show $\sum_{x=1}^\infty P(X=x)=1$, we can employ the identity:

$\sum_{x=1}^\infty P(X=x)$ = $p\sum_{x=1}^\infty (1-p)^{x-1}$ = $p$$1\over{1-(1-p)}$ = $p\over{p}$ = $1$.