5
$\begingroup$

Here are two popular principles in Statistics:

1) Law of Large Numbers: If $X$ is a random variable with a probability density function $f(x)$ and an expected value $E[X] = \mu$. If we take a sample of $n$ independent and identically distributed observations from $X$, denoted as $X_1, X_2, ..., X_n$, and calculate the sample average $\bar{X_n} = \frac{1}{n}\sum_{i=1}^{n}X_i$, then as $n$ becomes larger, $\bar{X_n}$ gets closer to $\mu$.

Mathematically, this is expressed as (Weak Law and Strong Law):

For any $\epsilon > 0$.

$$\lim_{n \to \infty} P(|\bar{X_n} - \mu| > \epsilon) = 0$$

$$P(\lim_{n \to \infty} \bar{X_n} = \mu) = 1$$

2) Bootstrap:

Let $X = \{x_1, x_2, ..., x_n\}$ be a sample of $n$ independent observations from an unknown probability distribution $F$ with mean $\mu$.

Let $\bar{X} = \frac{1}{n}\sum_{i=1}^{n}x_i$ be the sample mean.

We generate $B$ bootstrap samples $X^*_1, X^*_2, ..., X^*_B$, each of size $n$, by sampling with replacement from $X$. For each bootstrap sample $X^*_b$, we calculate its mean $\bar{X^*}_b$.

Then, the mean of these bootstrap sample means $\bar{X^*} = \frac{1}{B}\sum_{b=1}^{B}\bar{X^*}_b$ converges in probability to the actual mean $\mu$ as $B$ approaches infinity.

Mathematically, this is expressed as:

For any $\epsilon > 0$:

$$\lim_{B \to \infty} P(|\bar{X^*} - \mu| > \epsilon) = 0$$

Based on these principles, I have the following question:

  • For certain types of distributions, does the Law of Large Numbers require a smaller value of $n$ to achieve the same result (e.g. "distance" to the true mean) compared to other distributions?
  • For certain types of distribution, does the Bootstrap require smaller number of Bootstrap samples to achieve the same result (e.g. "distance" to the true mean) compared to other distributions?

As an example, different types of distributions could include:

  • Symmetric Distributions (e.g. Normal), Skewed Distributions (e.g. Exponential) and Multimodal Distributions (e.g. Mixture Distributions, Tweedie Distribution).
  • We can also categorize distributions based on their relative variances (e.g. a Normal Distribution where $\frac{\sigma}{\mu} = 2$ vs $\frac{\sigma}{\mu} = 10$).
  • We could also look at the dimensionality of the distributions (e.g. a univariate normal distribution, a bivariate normal distribution, a bivariate normal distribution with with a specific correlation structure, etc.) - does LLN/Bootstrap for lower dimension distribution require fewer samples to estimate the mean vector at the same level of precision compared to higher dimension distributions?

Theoretically, I would think that the Law of Large Numbers and Bootstrap would apply to all distributions (provided iid sampling is done correctly) - but is it possible that they might work "better/faster" in some types of distributions compared to other types of distributions (e.g. I think they might work better in symmetric distributions compared to non-symmetric distributions)?

Is there some way to quantify this bias-sample size trade off? (e.g. strength/speed of convergence)

$\endgroup$
6
  • $\begingroup$ I suggest splitting this into two questions: one about LLN and the other about bootstrap. In my view they are distinct enough to each deserve a separate thread. $\endgroup$ Commented Feb 27 at 13:15
  • $\begingroup$ The bootstrap $\bar{X}^*_B$ will converge to $\bar{X}$ when B goes to infinity, not $\mu$. That is because the bootstrap mean will converge to the mean of the empirical distribution $$F_n = \frac{1}{n}\sum_{i=1}^n \delta _{x_i}$$ which has mean $\bar{X}$ not $\mu$ $\endgroup$
    – Felix B.
    Commented Feb 27 at 15:11
  • $\begingroup$ Large Deviation theory might be of interest: en.wikipedia.org/wiki/Large_deviations_theory $\endgroup$
    – Felix B.
    Commented Feb 27 at 15:14
  • $\begingroup$ @ Richard: Should I make two separate questions? $\endgroup$ Commented Feb 27 at 16:07
  • $\begingroup$ @ Felix: Thanks for pointing this out ... I think I see the mistake I made: infinite bootstrap samples will converge to the population mean, not the theoretical mean of the random variable $\endgroup$ Commented Feb 27 at 16:08

1 Answer 1

7
$\begingroup$

The answer to the first question is yes and no and yes

First, yes: the variance of $X$ matters. For example, if the variance of $X$ (call it $\sigma^2$) is finite, $\bar X_n-\mu_n$ has variance $\sigma^2/n$. The 'typical' distance between $\bar X_n$ and $\mu$ is $\sigma/\sqrt{n}$: in fact, $\bar X_n$ has approximately a $N(\mu, \sigma^2/n)$ distribution.

Now, the 'no'. As long as $\sigma^2$ is finite, the typical distance between $\bar X_n$ and $\mu$ is $\sigma/\sqrt{n}$ and $\bar X_n$ has approximately a $N(\mu, \sigma^2/n)$ distribution irrespective of the distribution of $X$

And 'yes' again: the approximation of the distribution of $\bar X_n$ by a Normal distribution is better or worse according to how far $X$ is from Normal. If $X$ is very skewed or heavy-tailed, it will still be true that the typical distance from $\bar X_n$ to $\mu$ is $\sigma/\sqrt{n}$, but there will be a larger chance of it being substantially more or less than the typical value.

That's all with finite variances. If you have a distribution with no finite variance then $\bar X_n$ will converge more slowly to $\mu$, and if you have no finite mean (eg the Cauchy distribution) then $\bar X_n$ need not converge to $\mu$ at all.

The same thing is basically true for the bootstrap, but we're usually more interested in the distance from the bootstrap distribution to the distribution of $\bar X$ than the distance from $\bar X^*$ to $\bar X$ or $\mu$.

$\endgroup$
3
  • $\begingroup$ Although only indirectly related to the LLN, one might invoke concentration inequalities like en.wikipedia.org/wiki/Concentration_inequality to motivate better approximations of the expectation via the mean depending on the distribution? $\endgroup$ Commented Feb 27 at 10:52
  • $\begingroup$ thanks Thomas! Is there a way to quantify the speed of convergence? $\endgroup$ Commented Feb 27 at 16:10
  • $\begingroup$ Various options are given at stats.stackexchange.com/questions/555450/… ; note for instance that Sanov's theorem gives a fairly explicit constant in the convergence bound. $\endgroup$
    – helloworld
    Commented Feb 27 at 22:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.