# Calculating significance of difference between 2 groups using bootstrap

I'm calculating the significance of the difference between 2 groups using the bootstrap method in the way I haven't found in the literature but anyway makes sense to me. Now I want to double-check if I'm not shooting myself in the knee.

We have 2 groups with n (n is usually >1M and distribution is skewed) samples:

x: [x1, x2, x3, ..., xn]
y: [y1, y2, y3, ..., yn]

Procedure: For N(=1000?) times repeat:
1. For both groups, get bootstrap samples of size n and calculate mean. (It can also be some other statistic.)
2. Calculate the difference between means (or some other statistic).


Now we have the distribution of differences. Can we make statements such: I'm 95% confident that the difference of means is between a and b? or 'Difference of means is statistically significant if 95% confidence interval doesn't include 0'?

• Why are you doing this? Why not a standard test? Aug 6 '19 at 8:54
• It gives you a distribution of differences and feels intuitive. And you don't need to make any special assumption about the data - I think. Aug 6 '19 at 9:01
• Another option would be a permutation test stats.stackexchange.com/questions/20217/…. Aug 6 '19 at 9:19
• You are able to construct a confidence interval at a given level based on bootstrap sampling. Aug 6 '19 at 15:06

You don't say anything about the populations from which your data were randomly sampled. If sample sizes are at least moderately large and the data do not show signs of severe skewness (e.g., far outliers on one one side of center), then a two sample t confidence interval is probably the simplest and best choice. If data are from continuous non-normal populations, then a two sample Wilcoxon (rank sum) procedure may provide a useful confidence interval (CI).

For example, suppose we have samples of sizes $$n_1 = 30$$ and $$n_2 = 20$$ from respective distributions $$\textsf{Gamma}(\text{shape}=5, \text{rate}=.2)$$ and $$\textsf{Gamma}(6, .4).$$

set.seed(2019);  n1=30;  n2=20
x1 = rgamma(n1, 5, .2);  x2 = rgamma(n2, 6, .4)
summary(x1);  sd(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
7.653  16.314  24.456  27.025  33.651  59.145
[1] 13.13563  # sample SD
summary(x2);  sd(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
5.847   9.563  14.717  14.795  19.022  26.130
[1] 6.076901


In in each sample, the sample mean and median are near each other. Boxplots show no outliers. Also, notches in the boxes are nonparametric CIs; non-overlapping CIs suggest that the two population medians are different.

boxplot(x1, x2, notch=T, col="skyblue2")


Especially knowing the populations, one might raise theoretical objections to either the 95% Welch t CI $$(6.67, 17.79)$$ or the 95% Wilcoxon CI $$(4.79, 17.11)$$. But we know less just looking at descriptions of the sample, and there are no strong warnings in those descriptions against using either kind of CI. Results from relevant R procedures are shown below. Neither CI includes $$0,$$ so one can conclude that the difference $$\delta = \mu_1 - \mu_2$$ is significantly different from $$0.$$

t.test(x1, x2)$conf.int [1] 6.673883 17.786275 attr(,"conf.level") [1] 0.95 wilcox.test(x1, x2, conf.int=T)$conf.int
[1]  4.785996 17.109673
attr(,"conf.level")
[1] 0.95


However, you specifically asked about making a bootstrap confidence interval for the difference in population means. Your interest in making a bootstrap CI may be that you want to have the experience doing one, or that doing a bootstrap is a class assignment. There are very many styles of bootstrap CIs; I will illustrate one of the simpler styles.

You seek a CI for $$\delta = \mu_1 - \mu_2,$$ which can be estimated by $$D = \bar X_1 - \bar X_2.$$ If you knew the distribution of $$D,$$ then you could use it to find bounds $$L$$ and $$U$$ such that

$$0.95 = P(L \le D - \delta \le U) = P(-U+D \le \delta \le -L+D),$$

so that a 95% CI for $$\delta$$ would be of the form $$(D-U, D-L).$$

Recall that our observed value of $$D$$ is $$D_{obs}= \bar X_1- \bar X_2 = 12.23,$$ which we use in the bootstrap procedure.

d.obs = mean(x1) - mean(x2);  d.obs
[1] 12.23008


Not knowing the distribution of $$D$$ we use bootstrapping to find useful approximations $$L^*$$ of $$L$$ and $$U^*$$ of $$U.$$ In order to do this we take a large number $$B = 2000$$ of samples of size 30 with replacement form x1 and then find their means $$A_1^*.$$ Similarly, we 're-sample' from x2 to find means $$A_2^*.$$ From these, we find a vector of 2000 quantities $$D^* = A_1^* - A_2^*.$$ We would like to have a 'bootstrap distribution' of $$D - \delta.$$ Because $$\delta$$ is unknown, we use $$D_{obs}$$ as a proxy.

Then we find $$L* \approx L$$ and $$U^* \approx U$$ by taking quantiles .025 and .975 of $$D^* - D_{obs}.$$ From there, the 95% bootstrap Ci for $$\delta$$ is of the form $$(D_{obs}-U^*, D_{obs}-L^*),$$ where $$D_{obs}$$ returns to its original role.

In the R code below, we use .b instead of $$*$$ to indicate quantities arising from re-sampling. The 95% bootstrap CI for $$\delta$$ is $$(6.99, 17.29).$$ This is called a nonparametric CI because the bootstrap procedure did not involve the assumption that data are normal.

set.seed(1234);  B = 2000
D.b = replicate(B, mean(sample(x1,n1,rep=T)) -
mean(sample(x2,n2,rep=T)) - d.obs)
d.obs - quantile(D.b, c(.975,.025))
97.5%      2.5%
6.991987 17.289737


Addendum (per revision of question and comment): I agree with @Dave, that the t.test should be OK for such a large sample--even if markedly skewed.

I do not have much experience bootstrapping huge samples, but the bootstrap works well in the example I show below, comparing large exponential samples with respective population means 100 and 102.

The 95% t CI is $$(1.016. 2.343)$$ and the 95% nonparametric bootstrap CI is $$(1.026, 2.344).$$

set.seed(807)
x1 = rexp(10^5, 1/102);  x2=rexp(10^6, 1/100)
t.test(x1, x2)$conf.int [1] 1.016061 2.343464 attr(,"conf.level") [1] 0.95 set.seed(1234); B = 2000 n1 = length(x1); n2 = length(x2) d.obs = mean(x1) - mean(x2) D.b = replicate(B, mean(sample(x1,n1,rep=T)) - mean(sample(x2,n2,rep=T))-d.obs) d.obs-quantile(D.b, c(.975,.025)) 97.5% 2.5% 1.02571 2.34406  • Thank you! Both of my samples are larger than 100k points and their distributions are skewed: values are only positive and have a long tail. This is why I hesitate to use t-test. Do you have any comments on that? Aug 7 '19 at 9:13 • @Dave's comment is confirmed by my Addendum to the Answer. Aug 7 '19 at 11:13 • Does it matter that you resampled from each group separately rather than resampled the full sample and computed a mean difference within each sample? In the former, the number in each group is fixed; in the latter, the groups may have more or less in a given bootstrap sample than in the original sample. – Noah Aug 12 '19 at 7:25 • Because the goal is a bootstrap CI of the difference between two population means, I can't see the logic in combining the two groups. (What if one population has much larger variance than the other?) // For a permutation test of the difference, I might simulate the null distribution by randomly re-assigning$n_1$of the$n_1+n_2$observations to Gp 1 and the remaining$n_2\$ to Gp 2; then computing whatever distance metric. Aug 12 '19 at 7:33

"Difference of means is statistically significant if 95% confidence interval doesn't include 0"

Significant at the $$\alpha=0.05$$-level. Do specify that. If you want a p-value, you can find how big of a confidence interval you need for an endpoint of your CI to just kiss 0. But CIs and hypothesis testing have a ice duality.

I have to agree with BruceET, though. You seem to be out-clevering yourself. You would use bootstrap if you want to test something weird like $$(\kappa_1^3 - \kappa_2^3)^{1/3}$$ ($$\kappa_1$$ and $$\kappa_2$$ are the kurtosis of distributions 1 and 2, respectively). Then you would do your process and make inferences. But don't overthink testing means. BruceET has it right.

• Thank you! I added details about my sample distributions (over 100k points and skewed). Would you add anything to that? Aug 7 '19 at 9:28
• @Miha With that many points, t-testing is probably safe if what you want to examine is a difference in means. If you want to examine something funky about the kurtosis, then another test (such as the bootstrap procedure you described) would apply, F-test for comparing the variances, etc.
– Dave
Aug 7 '19 at 10:05