Bootstrapping and hypothesis testing

Question

I got a comment on a paper that I recently submitted.

He said, "Pag 7: referring to the “Univariate Analysis” section, bootstrap is not mentioned. This technique is extremely useful when dealing with datasets having few observations compared to the number of features, as often happens for metabolomic data. Is bootstrap performed by Metabox? If not, this should be highlighted since it is a limitation."

At page 7, I wrote, "• Univariate Analysis. Metabox collects a variety of well-established statistical hypothesis testing methods and post hoc analysis procedures (Table 1). In addition to the hypothesis testing procedures provided, metabox includes corresponding non-parametric testing procedures, post hoc analysis with false discovery rate (FDR) correction on both main effect level and simple main effect level, and power analyses at entity-level. Furthermore, metabox automatically and appropriately suggests statistical analysis methods according to the user-input study design. This feature aims to aid users through the depths of statistical terminology."

In the univariate analysis part, it is just a basic hypothesis testing procedure, including t-test and ANOVA, etc.

My question is that how to implement bootstrapping to, for example, t-test?

To my understanding, the bootstrap is just a resampling with replacement, which can be used to estimate the variation of a statistic. But when we do a hypothesis testing, does it make sense to estimate the variation of, for example, t (or p-value)?

Or do I misunderstand the comment?

Thanks a lot.

Answer 1

if you have data $X_1,\ldots,X_n$ and some test statistic $T(X_1,\ldots,X_n)$ you can do inference on the test statistic by using bootstrap. Resample (with replacement) your data $B$ times ($B$ large). At each iteration you will have a new data: $(X_{b_1},\ldots,X_{b_n})$. Calculate your test statistic for this data $T_b = T(X{b_1},\ldots,X_{b_n})$. Then you can estimate distributional quantities of your test statistic by using $T_b, b=1,\ldots,B$ as a sample from the test statistic distribution. This includes a $p$-value $$ p = \dfrac{1}{B}\sum_{b=1}^B \mathbf{1}\left\{ T_b > T\right\} $$ where $T$ is your test statistic of your original data. As Repmat noted in the comments, bootstrap is not a panacea for small sample size (see bootstrap disadvantages).

Answer 2

You're correct that bootstrapping is usually done to estimate confidence intervals, or more generally (properties of) the sampling distribution, of a statistic. For hypothesis testing, permutation tests are more common. The two are similar in certain aspects (e.g. both involve estimating a sampling distribution by using your existing data to create new samples, only the permutation test estimates the distribution of your statistic under a null hypothesis), so perhaps this reviewer did not fully understand or appreciate the difference? See also this question for more info on bootstrap vs. permutation.