I got a comment on a paper that I recently submitted.

He said, "Pag 7: referring to the “Univariate Analysis” section, bootstrap is not mentioned. This technique is extremely useful when dealing with datasets having few observations compared to the number of features, as often happens for metabolomic data. Is bootstrap performed by Metabox? If not, this should be highlighted since it is a limitation."

At page 7, I wrote, "• Univariate Analysis. Metabox collects a variety of well-established statistical hypothesis testing methods and post hoc analysis procedures (Table 1). In addition to the hypothesis testing procedures provided, metabox includes corresponding non-parametric testing procedures, post hoc analysis with false discovery rate (FDR) correction on both main effect level and simple main effect level, and power analyses at entity-level. Furthermore, metabox automatically and appropriately suggests statistical analysis methods according to the user-input study design. This feature aims to aid users through the depths of statistical terminology."

In the univariate analysis part, it is just a basic hypothesis testing procedure, including t-test and ANOVA, etc.

My question is that how to implement bootstrapping to, for example, t-test?

To my understanding, the bootstrap is just a resampling with replacement, which can be used to estimate the variation of a statistic. But when we do a hypothesis testing, does it make sense to estimate the variation of, for example, t (or p-value)?

Or do I misunderstand the comment?

Thanks a lot.

  • 2
    $\begingroup$ Actually there about as many variations on the bootstrap as you can possible imagine... Typically you bootstrap the t statistic, and use the distribution instead of the theoretical t distribution. I would like to add that the bootstrapping itself does not save you from small sample problems, as the reviewer appears to think. Can you provide more context? $\endgroup$
    – Repmat
    Dec 2, 2016 at 17:56
  • $\begingroup$ @Repmat Thanks for quick response. I added some more context from the reviewer. $\endgroup$
    – Frank Fan
    Dec 2, 2016 at 18:00
  • $\begingroup$ Could you say a bit more about specifics of your study: how many samples, numbers of experimental treatments, numbers of metabolites measured, and so on? Those details can make a difference. $\endgroup$
    – EdM
    Dec 2, 2016 at 18:57
  • $\begingroup$ This Cross Validated page also addresses the inability of bootstrapping to overcome small sample sizes; its advantage is not needing to make assumptions about distributions. Permutation tests might be more useful in some circumstances. $\endgroup$
    – EdM
    Dec 2, 2016 at 19:44

2 Answers 2


if you have data $X_1,\ldots,X_n$ and some test statistic $T(X_1,\ldots,X_n)$ you can do inference on the test statistic by using bootstrap. Resample (with replacement) your data $B$ times ($B$ large). At each iteration you will have a new data: $(X_{b_1},\ldots,X_{b_n})$. Calculate your test statistic for this data $T_b = T(X{b_1},\ldots,X_{b_n})$. Then you can estimate distributional quantities of your test statistic by using $T_b, b=1,\ldots,B$ as a sample from the test statistic distribution. This includes a $p$-value $$ p = \dfrac{1}{B}\sum_{b=1}^B \mathbf{1}\left\{ T_b > T\right\} $$ where $T$ is your test statistic of your original data. As Repmat noted in the comments, bootstrap is not a panacea for small sample size (see bootstrap disadvantages).


You're correct that bootstrapping is usually done to estimate confidence intervals, or more generally (properties of) the sampling distribution, of a statistic. For hypothesis testing, permutation tests are more common. The two are similar in certain aspects (e.g. both involve estimating a sampling distribution by using your existing data to create new samples, only the permutation test estimates the distribution of your statistic under a null hypothesis), so perhaps this reviewer did not fully understand or appreciate the difference? See also this question for more info on bootstrap vs. permutation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.