Why are hypothesis tests still used when we have the bootstrap and central limit theorem?

Question

Why are hypothesis tests still used when we have the bootstrap and central limit theorem?

To give context to my question, I briefly go over the central limit theorem and illustrate a simulation example using the R programming language.

The Wikipedia page for the central limit theorem provides some very good explanations of this theorem:

If $X_1, X_2,..., X_N$ are $n$ random samples drawn from a population with overall mean $\mu$ and finite variance $\sigma^2$, and if $\bar{X}_n$ is the sample mean, then the limiting form of the distribution, $Z=\lim_{n\to+\infty}\sqrt{n}\Big(\frac{\bar{X}_n-\mu}{\sigma}\Big)$, is a standard normal distribution.

I understand this as follows:

1) Take many random samples from any distribution

2) For each of these random samples, calculate their mean

3) The distribution of these means will follow a normal distribution (this result is particularly useful for inferences, e.g. hypothesis testing and confidence intervals).

I tried to see if I correctly understood the central limit theorem by creating two examples using the R programming language. I simulated non-normal data and took random samples from this data, in an attempt to view the "bell curved shape" corresponding to the distribution of these random samples.

1) Non-parametric bootstrap

In this example, imagine there is a town and you are interested in the salaries earned by people living in this town: specifically, you are interested in knowing if 20% of the population earns more than US$80,000.00. Here is the distribution for salaries in this town (in real life, you would not know how this distribution looks like - you could only take samples from this distribution):

set.seed(123)
a = rnorm(100000, 20000, 1000)

a2 = rnorm(100000, 40000, 10000)

a1 = rnorm(100000, 100000, 10000)

salary = c(a, a1, a2)
id = 1:length(salary)
my_data = data.frame(id, salary)


###plot

par(mfrow=c(1, 2))
hist(my_data$salary, 1000, ylab = "Number of People", xlab = " Salary ", main="Distribution for the Salaries of All People in Some Town: 300,000 People")
hist(our_sample$salary, 1000, ylab = "Number of People", xlab = " Salary ", main="Sample of the Full Data that we have Access to: 15,000 People")

Suppose we have access to the salaries of 5% of the people in this town (let's assume these are randomly chosen):

library(dplyr)
our_sample <- sample_frac(my_data, 0.05)

Next, we will take 1000 random samples from the 5% of this population we access to, and check the proportion of how many citizens earn more than US$80,000. I will then plot the distribution of these proportions - if I have done this correctly, I should expect to see a "bell curve shape":

library(dplyr)

results <- list()
    for (i in 1:1000) {

        train_i <- sample_frac(our_sample, 0.70)
        sid <- train_i$row
        train_i$prop = ifelse(train_i$salary >80000, 1, 0)

        results[[i]] <- mean(train_i$prop)
    }

results

results_df <- do.call(rbind.data.frame, results)

colnames(results_df)[1] <- "sample_mean"

hist(results_df$sample_mean)

As we can see, the original data was clearly non-normal, but the distribution for the mean of random samples from this non-normal data appears to look "somewhat normal":

par(mfrow=c(1, 3))
hist(my_data$salary, breaks = 10000, main = "full data")
hist(our_sample$salary, breaks = 10000, main = "sample of the full data that we have access too")
hist(results_df$sample_mean, breaks = 500, main = "resampled data by bootstrap from the data we hace access to")

Confidence Intervals:

• In reality, 32.5% of all citizens in this town earn more than US$80,000

  my_data$prop = ifelse(my_data$salary > 80000, 1, 0)
        mean(my_data$prop)
  
        [1] 0.3258367
  

• Surprisingly, according to the resampled bootstrap data, only 32.5% of the citizens earn more than US$80,000

  mean(results_df$sample_mean)
    [1] 0.3259046
  

The confidence interval is calculated as follows:

    results_df$delta = abs(mean(results_df$sample_mean) - results_df$sample_mean)

sorted_results = results_df[order(- results_df$delta), ]

quantile(sorted_results$delta,  probs = c(0.1, 0.9))

        10%         90%
0.000495400 0.005977933

This means that the confidence interval for the the proportion of citizens who earn more more than US$80,000 is between 32.59% - but in fact can be anywhere between (32.59 - 0.0495400 %) and (32.59 - 5.97%)

Conclusion: As far as I understand, the central limit theorem states that for any distribution, the distribution of the means from random samples will still follow a normal distribution. Furthermore, the non-parametric bootstrap also allows you to evaluate population inference and confidence intervals - regardless of the population's true distribution. Thus, why do we still use classical hypothesis testing methods? The only reason I can think of, is when there are smaller sample sizes. But are there any other reasons?

References

• Bootstrap confidence intervals. Class 24, 18.05. Jeremy Orloff and Jonathan Bloom.

Answer 1

Hypothesis tests are still used because they are motivated by a different need in statistical inference than interval estimators are motivated by.

The purpose of a hypothesis test is to make a decision as to whether there is evidence for the alternative hypothesis' expression of the population parameter.

Confidence intervals serve a different purpose: they provide a plausible range of estimates of a population parameter.

All the technical details about estimation (exact vs approximate, bootstrap vs non-probabilistic closed-form estimators, intervals, test statistics and p values, etc.) aside, the above represent fundamentally different motivations in statistical inference.

Aside: sometimes confidence interval coverage has a pretty direct correspondence to a hypothesis tests (a la does it cover the null or not), but this is not always the case, and habitually using confidence intervals for that purpose, in my opinion, obscures the above distinction between "let's make a decision about evidence for the alternative hypothesis" and "let's estimate a plausible range of values for a parameter."

Answer 2

One reason to use traditional hypothesis testing methods (when they can be used) is that it is computationally efficient to do so compared to bootstrap sampling. Depending upon the number of dimensions in your data, the number of bootstrap samples required to estimate p values (or confidence intervals) can be very large.

Central limit theorem is not always applicable. Sure, the average of a large number of i.i.d. random variables will lead to a Normal distribution. The question is what is large. Also, the problem is that it is not just the mean of the population you are interested in; there are other parameters that you want to estimate where CLT is not applicable. Again, we have asymptotic Normality to rescue (I would not go into details here), but it also requires a large sample. Again, what is large. Note that asymptotic Normality requires other technical conditions to hold which do not always hold.

Edit: An example where CLT is not applicable is when a time series has long-term persistence which means that autocorrelation dies very slowly. Here the assumption of independence is violated to the extent that CLT is not even approximately valid with thousands of samples. Here again you will have t resort to classic sampling distributions for hypothesis testing.

Another point (as detailed nicely by Alexis) is that the hypothesis tests are used for rejecting a plausible explanation (model) of the observed phenomenon. Therefore, hypothesis testing itself will stay relevant whatever method is used to test the hypothesis.

Answer 3

Let's see what actually happened with your example

• You started with a mixture of three normal distributions, where the probability of exceeding $\$80k$ was about $0.32576$
• You used this to construct a population of $300000$ with $97751$ cases exceeding $\$80k$, a proportion of about $0.32587$
• You sampled $15000$ without replacement from the $300000$ population with (my running of your code) $4865$ cases exceeding $\$80k$, a proportion of about $0.3243$. This is the natural estimate $\hat p$ of the proportion both of the population and of the original mixture distribution.
• You then $1000$ times sampled $10500$ without replacement from the $15000$ sample and looked at the cases exceeding $\$80k$ ranging from $3317$ to $3491$ with an average about $3405.8$, so getting proportions ranging from $0.3159$ to $0.3325$ with an average of about $0.3244$ (you seem to report a mean of $0.3259$ but I do not get that running your code - with only $1000$ repetitions the difference is not significant)

That last step is not conventional bootstrapping, which instead would involve sampling $15000$ (or perhaps $285000$) with replacement from the original sample, and doing it many more times. No matter. The real point is that nothing you do after the third step can improve on the knowledge that $4865$ of the $15000$ sample exceeded $\$80k$. What bootstrapping does do is allow estimates of statistics which are theoretically too difficult to calculate, but that is not the case with proportions.

In particular you cannot ensure you improve on $\hat p$ by bootstrapping, though there are some theoretical arguments for using the biased estimates $\frac{4865+\frac12}{15000+1}$ or $\frac{4865+1}{15000+2}$ or $\frac{4865+2}{15000+4}$ instead of $\frac{4865}{15000}$. As for a confidence interval, there are many theoretical suggestions, at least for the original mixture distribution, and here bootstrapping will not improve on these (and will vary since each set of bootstraps will vary). It is easy enough to extend these to confidence intervals for the population here, remembering that the population is finite and you already know $15000$ of the values.

Answer 4

Furthermore, the Non-Parametric Bootstrap also allows you to evaluate population inference and confidence intervals - regardless of the population's true distribution. Thus, why do we still use classical hypothesis testing methods? The only reason I can think of, is when there are smaller sample sizes. But are there any other reasons?

If you have a large sample then you can estimate the population very well based on the empirical distribution.

But instead of bootstrapping it can be done much faster with a simple computation. If you observe $p = 0.3258367$ in a sample of size $n = 15000$ then the estimate for the standard error in the estimate $\hat{p}$ is

$$\sigma_{\hat{p}} \approx \sqrt{\frac{p(1-p)}{n}}$$

and for the 80% confidence interval

$$CI = \hat{p} \pm 1.281552 \sigma_{\hat{p}} $$

giving

$$CI = 0.3258367 \pm 0.004904255 $$

Advantage direct computation / disadvantage bootstrap

• So this computation did not require all the resampling that you needed to do.

• It is also more accurate than the result that you give*.

• In addition, the bootstrapping is a black box. It gives you only a single output but there is no information about deeper relationships.

  For instance, with the formula for the direct computation you can see the $1/\sqrt{n}$ dependence of the end result. This is useful if you want to determine the best sample size for a certain precision. With bootstrapping you would not be able to see this directly and you would have to simulate many situations.

---

*some errors must have slipped into your computation since "between (32.59 - 0.0495400 %) and (32.59 - 5.97%)" is wrong. You should get something like (32.59 - 0.49 %) and (32.59 + 0.49 %), so the correct interval size is only about 1% in size.

Answer 5

The bootstrap cannot resample events that didn't occur in the dataset. If the probability of some event was very low and ends up not occuring -- e.g., the expected number of events in some range or bin is less than 1, and indeed zero are obtained -- the bootstrap procedure will of course never be able to produce (re)samples with a non-zero number of such events. In other words, as opposed to sampling from the true underlying distribution, the bootstrap implicitly assumes that the probability of this event was strictly zero. This can lead to biases in the case of sparse data or long-tailed data.

Of course, sparse data can also be a problem in hypothesis tests, especially those that assume an asymptotic distribution for some underlying metric, and special care is likely to be needed.

Answer 6

1. Take many random samples from any distribution

But your example only has a single sample that you're resampling. You are not taking samples from the population, you are resampling your sample, so you actually just evaluate your sample's parameters that you could've found directly.

3. The distribution of thee means will follow a Normal Distribution (this result is particularly useful for inferences, e.g. hypothesis testing and confidence intervals).

You are missing the fact that the central limit theorem requires $n \rightarrow \infty$. For the finite case you get something else. If the population follows normal distribution, means will follow Student's t-distribution.

P.S. Please don't take my direct language as an attack. The question is great and it's important to discuss these topics. I just felt these misunderstandings should be countered explicitly and concisely.

Answer 7

1. There are scenarios, though I will grant that many of them are a bit esoteric, where the bootstrap is not consistent. For instance: parameters involving ranks, or when a parameter is at the boundary of a space.
2. The bootstrap can be tricky, or inefficient to implement with non-iid data. For instance, with clustered data, one is forced to resample the highest-level of clustering.
3. To get efficient estimates of confidence intervals, under the hood of many bootstrap estimators is something that looks like a classical estimator, for instance the studentized bootstrap. Of note, one can't just naively take the 2.5% and 97.5% percentile of the bootstrapped statistic in the resampled data and expect good coverage.

Answer 8

Classical methods, designed experiments analyzed with analysis of variance, were designed to work with small samples and complex design structure. Complex design structure includes Latin Squares, Balanced Incomplete Block Designs, Repeated Measures Designs, and others. All of these are used regularly in biological and psychological sciences and chemical engineering.

The original method of analysis, analysis of variance, gives tests of hypotheses and, for the complex designs gives descriptive information that is useful in understanding the scientific conclusions, eg, interval estimates, identification of outliers or failures of assumptions. Analysis of Variance also works well for randomized designs even when there are some deviations from the assumptions. There is a large literature from the 1950s that shows that randomization is a generally good basis for the analysis of variance tests, even though most derivations of tests are based on Gaussian distributions.

Methods for the analysis of binomial and multinomial measurements were developed for many of the designs scientists used and analyzed, perhaps somewhat improperly, with Analysis of Variance. The experimental designs are still in common use with appropriate analyses.

The Bootstrap is a valuable tool. The Bootstrap works with as well with small samples as well as large. The Bootstrap gives estimates of bias and explicit methods that can be used to reduce bias. While most analysis with the bootstrap has focused on estimation there is no reason it can't be used for tests of hypotheses. It also works will with measurements that are somewhat non-normal.

The Bootstrap does not work so well with the complex designs such as Randomized Block Designs, Greco-Latin Squares, etc. I would be happy to see sample reuse methods, the Bootstrap or otherwise, described for such designs. Until it is the classical methods of analysis will continue to be useful. I won't hold my breath waiting.