I came across this page and was surprised and confused by the following:

A naive person attempting to do a bootstrap test just calculates a P-value as something like

mean(tstat.star > tstat.hat)

where tstat.hat is the value of the test statistic calculated for the actual data and tstat.star is a vector of values of the test statistic calculated for bootstrap samples.

The resulting test is a valid hypothesis test in the sense that a test with nominal significance level α actually has that significance level.

But (a big but!) this test typically has no power. It rejects the null hypothesis with probability α regardless of what the alternative is. No matter how large the deviation of the true parameter value from the null hypothesis, the naive bootstrap test typically doesn't find any "statistical significance."

He goes on to say

There are a variety of special situations in which something that makes sense as a nonparametric bootstrap hypothesis test can be done. Efron and Tibshirani, Chapter 16 describe a few. But if you don't see a general principle in their explanation, don't worry. There isn't any.

I'm having a hard time coming up with an example in which the test has no power, regardless of the true parameter value. Also, based on the second quote, I suppose the standard examples for nonparametric hypothesis testing are considered by Geyer to be special situations?

I understand that for tests that don't involve just shuffling or otherwise manipulating data labels, it can be difficult to sample from the null hypothesis, but it seems that Geyer is already assuming a valid test, i.e., this is not the failure mode he has in mind.

  • 1
    $\begingroup$ I think the key here is that he's looking at, e.g., bootstrapped t-statistics, and the test he's criticizing is one where you compare the bootstrapped t-statistics with the observed (full-sample) t-statistic, which of course won't work; you'd need to compare the bootstrapped t-statistics to the expected t-statistic under the null hypothesis, which would almost always be zero (I can't think of a situation where it wouldn't be, but there probably is one.) This logic would apply for other test statistics as well. $\endgroup$
    – jbowman
    Nov 9, 2022 at 3:29
  • $\begingroup$ ... e.g., "My t-statistic is 116. Now I'll construct a bootstrap... the bootstrap 95% CI is 113-118, and 116 is between 113 and 118, so I can't reject the null at the 95% level of confidence." $\endgroup$
    – jbowman
    Nov 9, 2022 at 3:33
  • $\begingroup$ Is the sentiment something like: "if you're bootstrapping just by drawing random samples from your observed distribution, of course your observed data's statistic is not going to be improbable wrt the bootstrapped distribution of statistics." But this seems like a complete misuse of the bootstrap. There needs to be some sampling 'trick' to break up the potential structure in the observed data. The correlation code he uses is a perfect example. Shouldn't he be drawing x and y using different random indices (a 'trick')?? Otherwise, we're just drawing noisy versions of our observed data... $\endgroup$ Nov 11, 2022 at 19:18
  • $\begingroup$ Power is the probability to reject the null hypothesis computed under a specific alternative hypothesis. The "naive bootstrap test" p = mean(tstar.star > tstat.hat) <= alpha = 0.05 typically doesn't reject. Since the probability to reject is close to 0, the power is close to 0. This doesn't say much other than the bootstrap doesn't fit naturally in the framework of null hypothesis significance testing. The bootstrap is a method for calculating confidence intervals. $\endgroup$
    – dipetkov
    Nov 19, 2022 at 23:06

1 Answer 1


Here is how I would interpret Geyer in a simulation:

X <- rnorm(20, mean = 2, sd = 4)

# test statistic calculated for the actual data  H0: mu = 0
tstat.hat <- (mean(X) - 0) / (sd(X) / sqrt(20))

# vector of values of the test statistic calculated for bootstrap samples
tstat.star <- numeric(100000)
for (i in seq_along(tstat.star))
  Xi <- sample(X, size = 20, replace = TRUE)
  tstat.star[i] <- (mean(Xi) - 0) / (sd(X) / sqrt(20))

# expect this to be near 0.5 for a symmetric distribution
#   Not sure why this is considered to be a valid hypothesis test with alpha significance
mean(tstat.star > tstat.hat)
#> [1] 0.49865

# this makes more sense as a valid hypothesis test  H0: mu < 0
mean(tstat.star <= 0)
#> [1] 0.10838
t.test(X, alternative = "greater")$p.value
#> [1] 0.1212443

# also agree with the confidence interval idea, of course
quantile(tstat.star, probs = c(0.025, 0.975))
#>       2.5%      97.5% 
#> -0.7010223  3.1392164
c(mean(X) + qt(0.025, df = 20-1)*sd(X)/sqrt(20), mean(X) + qt(0.975, df = 20-1)*sd(X)/sqrt(20))
#> [1] -0.8450756  3.1445932


# if null is true H0: mu = 0

X <- rnorm(20, mean = 0, sd = 4)

tstat.hat <- (mean(X) - 0) / (sd(X) / sqrt(20))

tstat.star <- numeric(100000)
for (i in seq_along(tstat.star))
  Xi <- sample(X, size = 20, replace = TRUE)
  tstat.star[i] <- (mean(Xi) - 0) / (sd(X) / sqrt(20))

# same
mean(tstat.star > tstat.hat)
#> [1] 0.4944

mean(tstat.star < 0)
#> [1] 0.4627

quantile(tstat.star, probs = c(0.025, 0.975))
#>      2.5%     97.5% 
#> -1.755945  2.046421

If you look down the page in the link, Geyer does the same type of thing:

x <- rnorm(20, 0, 1)
y <- rnorm(20, 0, 1)

theta.hat <- cor(x, y)

n <- length(x)
nboot <- 1e5
theta.star <- double(nboot)
for (i in 1:nboot) {
  k.star <- sample(n, replace = TRUE)
  theta.star[i] <- cor(x[k.star], y[k.star])

## lower-tailed test of theta = 0
ltpv <- mean(theta.star <= 0)
#> [1] 0.95744

## upper-tailed test of theta = 0
utpv <- mean(theta.star >= 0)
#> [1] 0.04256

## two-tailed test of theta = 0
2 * min(ltpv, utpv)
#> [1] 0.08512

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.