27
$\begingroup$

Why are hypothesis tests still used when we have the bootstrap and central limit theorem?

To give context to my question, I briefly go over the central limit theorem and illustrate a simulation example using the R programming language.

The Wikipedia page for the central limit theorem provides some very good explanations of this theorem:

If $X_1, X_2,..., X_N$ are $n$ random samples drawn from a population with overall mean $\mu$ and finite variance $\sigma^2$, and if $\bar{X}_n$ is the sample mean, then the limiting form of the distribution, $Z=\lim_{n\to+\infty}\sqrt{n}\Big(\frac{\bar{X}_n-\mu}{\sigma}\Big)$, is a standard normal distribution.

I understand this as follows:

1) Take many random samples from any distribution

2) For each of these random samples, calculate their mean

3) The distribution of these means will follow a normal distribution (this result is particularly useful for inferences, e.g. hypothesis testing and confidence intervals).

Illustration of central limit theorem leading to Gaussian distribution of sample mean

I tried to see if I correctly understood the central limit theorem by creating two examples using the R programming language. I simulated non-normal data and took random samples from this data, in an attempt to view the "bell curved shape" corresponding to the distribution of these random samples.

1) Non-parametric bootstrap

In this example, imagine there is a town and you are interested in the salaries earned by people living in this town: specifically, you are interested in knowing if 20% of the population earns more than US$80,000.00. Here is the distribution for salaries in this town (in real life, you would not know how this distribution looks like - you could only take samples from this distribution):

set.seed(123)
a = rnorm(100000, 20000, 1000)

a2 = rnorm(100000, 40000, 10000)

a1 = rnorm(100000, 100000, 10000)

salary = c(a, a1, a2)
id = 1:length(salary)
my_data = data.frame(id, salary)


###plot

par(mfrow=c(1, 2))
hist(my_data$salary, 1000, ylab = "Number of People", xlab = " Salary ", main="Distribution for the Salaries of All People in Some Town: 300,000 People")
hist(our_sample$salary, 1000, ylab = "Number of People", xlab = " Salary ", main="Sample of the Full Data that we have Access to: 15,000 People")

Enter image description here

Suppose we have access to the salaries of 5% of the people in this town (let's assume these are randomly chosen):

library(dplyr)
our_sample <- sample_frac(my_data, 0.05)

Next, we will take 1000 random samples from the 5% of this population we access to, and check the proportion of how many citizens earn more than US$80,000. I will then plot the distribution of these proportions - if I have done this correctly, I should expect to see a "bell curve shape":

library(dplyr)

results <- list()
    for (i in 1:1000) {

        train_i <- sample_frac(our_sample, 0.70)
        sid <- train_i$row
        train_i$prop = ifelse(train_i$salary >80000, 1, 0)

        results[[i]] <- mean(train_i$prop)
    }

results

results_df <- do.call(rbind.data.frame, results)

colnames(results_df)[1] <- "sample_mean"

hist(results_df$sample_mean)

Enter image description here

As we can see, the original data was clearly non-normal, but the distribution for the mean of random samples from this non-normal data appears to look "somewhat normal":

par(mfrow=c(1, 3))
hist(my_data$salary, breaks = 10000, main = "full data")
hist(our_sample$salary, breaks = 10000, main = "sample of the full data that we have access too")
hist(results_df$sample_mean, breaks = 500, main = "resampled data by bootstrap from the data we hace access to")

Enter image description here

Confidence Intervals:

  • In reality, 32.5% of all citizens in this town earn more than US$80,000

    my_data$prop = ifelse(my_data$salary > 80000, 1, 0)
          mean(my_data$prop)
    
          [1] 0.3258367
    
  • Surprisingly, according to the resampled bootstrap data, only 32.5% of the citizens earn more than US$80,000

    mean(results_df$sample_mean)
      [1] 0.3259046
    

The confidence interval is calculated as follows:

Enter image description here

    results_df$delta = abs(mean(results_df$sample_mean) - results_df$sample_mean)

sorted_results = results_df[order(- results_df$delta), ]

quantile(sorted_results$delta,  probs = c(0.1, 0.9))

        10%         90%
0.000495400 0.005977933

This means that the confidence interval for the the proportion of citizens who earn more more than US$80,000 is between 32.59% - but in fact can be anywhere between (32.59 - 0.0495400 %) and (32.59 - 5.97%)

Conclusion: As far as I understand, the central limit theorem states that for any distribution, the distribution of the means from random samples will still follow a normal distribution. Furthermore, the non-parametric bootstrap also allows you to evaluate population inference and confidence intervals - regardless of the population's true distribution. Thus, why do we still use classical hypothesis testing methods? The only reason I can think of, is when there are smaller sample sizes. But are there any other reasons?

References

$\endgroup$
6
  • 2
    $\begingroup$ A related (closed) question is here: Is Statistics less important in the era of big data than in old days?. I believe the topic (why not just gather more data) has occured somewhere else as well. $\endgroup$ Oct 23 at 9:40
  • 10
    $\begingroup$ The central limit theorem is for statisticians, not for the application of statistics. And the bootstrap is an approximate method; in many cases the approximation is not very accurate. The big question is why do we test hypotheses instead of estimating quantities of interest? $\endgroup$ Oct 23 at 14:05
  • 1
    $\begingroup$ "As we can see, the original data was clearly non-normal but the distribution for the mean of random samples from this non-normal data appears to look "somewhat normal"" The histogram is not the distribution for the mean of random samples. Instead those values between 0.315 and 0.330 are the fractions of the random samples that earn more than 80 grand. $\endgroup$ Oct 23 at 15:40
  • 5
    $\begingroup$ "Central Limit Theorem states that for any distribution" -- citation needed. In fact, the CLT does not state this; the CLT has nontrivial hypotheses. For instance, the CLT does not apply to the Cauchy distribution. Likewise, CLT does not apply to other distributions whose variances do not exist. $\endgroup$ Oct 24 at 15:14
  • 1
    $\begingroup$ Can you fix the indentation of the example code (e.g., near for (i in 1:1000) {) and otherwise format it consistently? Thanks in advance for your kind consideration of your readers. $\endgroup$ Oct 25 at 14:28
25
$\begingroup$

Hypothesis tests are still used because they are motivated by a different need in statistical inference than interval estimators are motivated by.

The purpose of a hypothesis test is to make a decision as to whether there is evidence for the alternative hypothesis' expression of the population parameter.

Confidence intervals serve a different purpose: they provide a plausible range of estimates of a population parameter.

All the technical details about estimation (exact vs approximate, bootstrap vs non-probabilistic closed-form estimators, intervals, test statistics and p values, etc.) aside, the above represent fundamentally different motivations in statistical inference.


Aside: sometimes confidence interval coverage has a pretty direct correspondence to a hypothesis tests (a la does it cover the null or not), but this is not always the case, and habitually using confidence intervals for that purpose, in my opinion, obscures the above distinction between "let's make a decision about evidence for the alternative hypothesis" and "let's estimate a plausible range of values for a parameter."

$\endgroup$
9
  • 2
    $\begingroup$ @cdalitz Do tell. Also: a symmantic quibble: your point is specific to null hypotheses on statistics with continuous distributions of the form $\text{H}_{0}\text{: }\theta = 0$ (or, for ratio tests $\text{H}_{0}\text{: }\theta = 1$), but not to those of the form $\text{H}_{0}\text{: }\theta \ge 0$, $\text{H}_{0}\text{: }\theta \le 0$, or $\text{H}_{0}\text{: }|\theta| \ge \Delta$. $\endgroup$
    – Alexis
    Oct 24 at 16:54
  • 1
    $\begingroup$ @cdalitz As a consequence "The null hypothesis is always known to be false" is true only for quite specific circumstances. $\endgroup$
    – Alexis
    Oct 24 at 16:58
  • 2
    $\begingroup$ @cdalitz Well, yes it is, and there are plenty of textbooks which disagree with you on that point (I suspect you express one sided test null and alternate hypotheses thus: $\text{H}_{0}\text{: }\theta = 0$, $\text{H}_{1}\text{: }\theta < 0$, which leaves unexpressed by either null or alternative the case where $\theta>0$?; I am in the camp which view the alternative strictly as the complement of the null, hence above notation). In my notation, '$\theta$' is whatever the population statistic of inferential interest is. For example, $\theta = \mu - \mu_0$, $\theta = \mu_1 - \mu_2$, etc. $\endgroup$
    – Alexis
    Oct 24 at 17:15
  • 2
    $\begingroup$ Not only is the null hypothesis always known to be false, also all parametric models are false and data is never i.i.d. With enough data surely not all tests will reject the null hypothesis. Try to reject $\theta=0$ with a t-test when the true underlying distribution is Cauchy! Tests are generally not about finding out which models are true (none are, anyway), but rather about whether the data contradict a certain idealised model. As things stand, sometimes they do and sometimes they don't, so tests are in fact informative. $\endgroup$ Oct 24 at 19:08
  • 2
    $\begingroup$ @ChristianHennig Surely All models are false. Some are useful.—of a Box vinatge $\endgroup$
    – Alexis
    Oct 24 at 19:18
14
$\begingroup$

One reason to use traditional hypothesis testing methods (when they can be used) is that it is computationally efficient to do so compared to bootstrap sampling. Depending upon the number of dimensions in your data, the number of bootstrap samples required to estimate p values (or confidence intervals) can be very large.

Central limit theorem is not always applicable. Sure, the average of a large number of i.i.d. random variables will lead to a Normal distribution. The question is what is large. Also, the problem is that it is not just the mean of the population you are interested in; there are other parameters that you want to estimate where CLT is not applicable. Again, we have asymptotic Normality to rescue (I would not go into details here), but it also requires a large sample. Again, what is large. Note that asymptotic Normality requires other technical conditions to hold which do not always hold.

Edit: An example where CLT is not applicable is when a time series has long-term persistence which means that autocorrelation dies very slowly. Here the assumption of independence is violated to the extent that CLT is not even approximately valid with thousands of samples. Here again you will have t resort to classic sampling distributions for hypothesis testing.

Another point (as detailed nicely by Alexis) is that the hypothesis tests are used for rejecting a plausible explanation (model) of the observed phenomenon. Therefore, hypothesis testing itself will stay relevant whatever method is used to test the hypothesis.

$\endgroup$
7
  • 3
    $\begingroup$ Using statistical (as opposed to biological) terminology, a large number of samples should be a large sample. $\endgroup$ Oct 23 at 8:45
  • 4
    $\begingroup$ It is a mystery nevertheless. In practical applications, where not only the population mean but parameters are required, 30 samples are nowhere near enough. For example, long term persistence in time-series analysis means that even thousands of samples are not enough for Normal approximation to be valid. Sure, you can assume i.id., then 30 samples are enough. I will add this point to the answer actually. $\endgroup$ Oct 23 at 17:57
  • 1
    $\begingroup$ It depends a lot on the field, @RichardHardy: I thought the 30 was a 30k. That is sometimes sufficient for us. Sometimes it's rather 30 mio. $\endgroup$
    – Mayou36
    Oct 24 at 13:34
  • 3
    $\begingroup$ @Mayou36, I understand. Since Cross Validated is a statistical site, my idea is to encourage the use of statistical terminology. (Though actually the idea is not mine; I have borrowed it from other users on the site.) $\endgroup$ Oct 24 at 15:55
  • 1
    $\begingroup$ No sorry, I may didn't express myself clearly: I meant I misread as 30k, because that would be the order of magnitude that I would argue in. This was no critiques to the notation, that is perfectly fine, it was merely strengthening the point that "sufficient" can be very different, depending a lot on the goals (such as which p-value? what do we want to extract?) and your data (such as dimensionality), i.e. on the field. $\endgroup$
    – Mayou36
    Oct 25 at 10:28
11
$\begingroup$

Let's see what actually happened with your example

  • You started with a mixture of three normal distributions, where the probability of exceeding $\$80k$ was about $0.32576$
  • You used this to construct a population of $300000$ with $97751$ cases exceeding $\$80k$, a proportion of about $0.32587$
  • You sampled $15000$ without replacement from the $300000$ population with (my running of your code) $4865$ cases exceeding $\$80k$, a proportion of about $0.3243$. This is the natural estimate $\hat p$ of the proportion both of the population and of the original mixture distribution.
  • You then $1000$ times sampled $10500$ without replacement from the $15000$ sample and looked at the cases exceeding $\$80k$ ranging from $3317$ to $3491$ with an average about $3405.8$, so getting proportions ranging from $0.3159$ to $0.3325$ with an average of about $0.3244$ (you seem to report a mean of $0.3259$ but I do not get that running your code - with only $1000$ repetitions the difference is not significant)

That last step is not conventional bootstrapping, which instead would involve sampling $15000$ (or perhaps $285000$) with replacement from the original sample, and doing it many more times. No matter. The real point is that nothing you do after the third step can improve on the knowledge that $4865$ of the $15000$ sample exceeded $\$80k$. What bootstrapping does do is allow estimates of statistics which are theoretically too difficult to calculate, but that is not the case with proportions.

In particular you cannot ensure you improve on $\hat p$ by bootstrapping, though there are some theoretical arguments for using the biased estimates $\frac{4865+\frac12}{15000+1}$ or $\frac{4865+1}{15000+2}$ or $\frac{4865+2}{15000+4}$ instead of $\frac{4865}{15000}$. As for a confidence interval, there are many theoretical suggestions, at least for the original mixture distribution, and here bootstrapping will not improve on these (and will vary since each set of bootstraps will vary). It is easy enough to extend these to confidence intervals for the population here, remembering that the population is finite and you already know $15000$ of the values.

$\endgroup$
1
  • 1
    $\begingroup$ thank you for your answer! At the end of the day, is what I have done correct? (With the exception of sampling with replacement)? Thank you! $\endgroup$
    – stats555
    Oct 24 at 20:02
8
$\begingroup$

Furthermore, the Non-Parametric Bootstrap also allows you to evaluate population inference and confidence intervals - regardless of the population's true distribution. Thus, why do we still use classical hypothesis testing methods? The only reason I can think of, is when there are smaller sample sizes. But are there any other reasons?

If you have a large sample then you can estimate the population very well based on the empirical distribution.

But instead of bootstrapping it can be done much faster with a simple computation. If you observe $p = 0.3258367$ in a sample of size $n = 15000$ then the estimate for the standard error in the estimate $\hat{p}$ is

$$\sigma_{\hat{p}} \approx \sqrt{\frac{p(1-p)}{n}}$$

and for the 80% confidence interval

$$CI = \hat{p} \pm 1.281552 \sigma_{\hat{p}} $$

giving

$$CI = 0.3258367 \pm 0.004904255 $$

Advantage direct computation / disadvantage bootstrap

  • So this computation did not require all the resampling that you needed to do.

  • It is also more accurate than the result that you give*.

  • In addition, the bootstrapping is a black box. It gives you only a single output but there is no information about deeper relationships.

    For instance, with the formula for the direct computation you can see the $1/\sqrt{n}$ dependence of the end result. This is useful if you want to determine the best sample size for a certain precision. With bootstrapping you would not be able to see this directly and you would have to simulate many situations.


*some errors must have slipped into your computation since "between (32.59 - 0.0495400 %) and (32.59 - 5.97%)" is wrong. You should get something like (32.59 - 0.49 %) and (32.59 + 0.49 %), so the correct interval size is only about 1% in size.

$\endgroup$
4
$\begingroup$

The bootstrap cannot resample events that didn't occur in the dataset. If the probability of some event was very low and ends up not occuring -- e.g., the expected number of events in some range or bin is less than 1, and indeed zero are obtained -- the bootstrap procedure will of course never be able to produce (re)samples with a non-zero number of such events. In other words, as opposed to sampling from the true underlying distribution, the bootstrap implicitly assumes that the probability of this event was strictly zero. This can lead to biases in the case of sparse data or long-tailed data.

Of course, sparse data can also be a problem in hypothesis tests, especially those that assume an asymptotic distribution for some underlying metric, and special care is likely to be needed.

$\endgroup$
3
$\begingroup$
  1. Take many random samples from any distribution

But your example only has a single sample that you're resampling. You are not taking samples from the population, you are resampling your sample, so you actually just evaluate your sample's parameters that you could've found directly.

  1. The distribution of thee means will follow a Normal Distribution (this result is particularly useful for inferences, e.g. hypothesis testing and confidence intervals).

You are missing the fact that the central limit theorem requires $n \rightarrow \infty$. For the finite case you get something else. If the population follows normal distribution, means will follow Student's t-distribution.

P.S. Please don't take my direct language as an attack. The question is great and it's important to discuss these topics. I just felt these misunderstandings should be countered explicitly and concisely.

$\endgroup$
2
$\begingroup$

Classical methods, designed experiments analyzed with analysis of variance, were designed to work with small samples and complex design structure. Complex design structure includes Latin Squares, Balanced Incomplete Block Designs, Repeated Measures Designs, and others. All of these are used regularly in biological and psychological sciences and chemical engineering.

The original method of analysis, analysis of variance, gives tests of hypotheses and, for the complex designs gives descriptive information that is useful in understanding the scientific conclusions, eg, interval estimates, identification of outliers or failures of assumptions. Analysis of Variance also works well for randomized designs even when there are some deviations from the assumptions. There is a large literature from the 1950s that shows that randomization is a generally good basis for the analysis of variance tests, even though most derivations of tests are based on Gaussian distributions.

Methods for the analysis of binomial and multinomial measurements were developed for many of the designs scientists used and analyzed, perhaps somewhat improperly, with Analysis of Variance. The experimental designs are still in common use with appropriate analyses.

The Bootstrap is a valuable tool. The Bootstrap works with as well with small samples as well as large. The Bootstrap gives estimates of bias and explicit methods that can be used to reduce bias. While most analysis with the bootstrap has focused on estimation there is no reason it can't be used for tests of hypotheses. It also works will with measurements that are somewhat non-normal.

The Bootstrap does not work so well with the complex designs such as Randomized Block Designs, Greco-Latin Squares, etc. I would be happy to see sample reuse methods, the Bootstrap or otherwise, described for such designs. Until it is the classical methods of analysis will continue to be useful. I won't hold my breath waiting.

$\endgroup$
1
$\begingroup$
  1. There are scenarios, though I will grant that many of them are a bit esoteric, where the bootstrap is not consistent. For instance: parameters involving ranks, or when a parameter is at the boundary of a space.
  2. The bootstrap can be tricky, or inefficient to implement with non-iid data. For instance, with clustered data, one is forced to resample the highest-level of clustering.
  3. To get efficient estimates of confidence intervals, under the hood of many bootstrap estimators is something that looks like a classical estimator, for instance the studentized bootstrap. Of note, one can't just naively take the 2.5% and 97.5% percentile of the bootstrapped statistic in the resampled data and expect good coverage.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.