# Bootstrapping and hypothesis testing

I got a comment on a paper that I recently submitted.

He said, "Pag 7: referring to the “Univariate Analysis” section, bootstrap is not mentioned. This technique is extremely useful when dealing with datasets having few observations compared to the number of features, as often happens for metabolomic data. Is bootstrap performed by Metabox? If not, this should be highlighted since it is a limitation."

At page 7, I wrote, "• Univariate Analysis. Metabox collects a variety of well-established statistical hypothesis testing methods and post hoc analysis procedures (Table 1). In addition to the hypothesis testing procedures provided, metabox includes corresponding non-parametric testing procedures, post hoc analysis with false discovery rate (FDR) correction on both main effect level and simple main effect level, and power analyses at entity-level. Furthermore, metabox automatically and appropriately suggests statistical analysis methods according to the user-input study design. This feature aims to aid users through the depths of statistical terminology."

In the univariate analysis part, it is just a basic hypothesis testing procedure, including t-test and ANOVA, etc.

My question is that how to implement bootstrapping to, for example, t-test?

To my understanding, the bootstrap is just a resampling with replacement, which can be used to estimate the variation of a statistic. But when we do a hypothesis testing, does it make sense to estimate the variation of, for example, t (or p-value)?

Or do I misunderstand the comment?

Thanks a lot.

• Actually there about as many variations on the bootstrap as you can possible imagine... Typically you bootstrap the t statistic, and use the distribution instead of the theoretical t distribution. I would like to add that the bootstrapping itself does not save you from small sample problems, as the reviewer appears to think. Can you provide more context? – Repmat Dec 2 '16 at 17:56
• @Repmat Thanks for quick response. I added some more context from the reviewer. – Frank Fan Dec 2 '16 at 18:00
• Could you say a bit more about specifics of your study: how many samples, numbers of experimental treatments, numbers of metabolites measured, and so on? Those details can make a difference. – EdM Dec 2 '16 at 18:57
• This Cross Validated page also addresses the inability of bootstrapping to overcome small sample sizes; its advantage is not needing to make assumptions about distributions. Permutation tests might be more useful in some circumstances. – EdM Dec 2 '16 at 19:44

if you have data $X_1,\ldots,X_n$ and some test statistic $T(X_1,\ldots,X_n)$ you can do inference on the test statistic by using bootstrap. Resample (with replacement) your data $B$ times ($B$ large). At each iteration you will have a new data: $(X_{b_1},\ldots,X_{b_n})$. Calculate your test statistic for this data $T_b = T(X{b_1},\ldots,X_{b_n})$. Then you can estimate distributional quantities of your test statistic by using $T_b, b=1,\ldots,B$ as a sample from the test statistic distribution. This includes a $p$-value $$p = \dfrac{1}{B}\sum_{b=1}^B \mathbf{1}\left\{ T_b > T\right\}$$ where $T$ is your test statistic of your original data. As Repmat noted in the comments, bootstrap is not a panacea for small sample size (see bootstrap disadvantages).