I have two small data sets (n < 15) that are not normally distributed. Each measurement in the two datasets was experimentally determined and has an uncertainty associated with it. I'd like to use a bootstrap method to estimate the mean and standard error of the two data sets so I can test if the means are significantly different from one another.

I know that bootstrapping draws subsamples directly from the measurements. However, if each measurement has an uncertainty associated with it, does bootstrapping take this into account? Or are there different methods that would do account for the uncertainty in the measurements (Monte Carlo, etc.)?

  • $\begingroup$ Why not a t-test and F-test? $\endgroup$ Commented Aug 21, 2019 at 11:37
  • $\begingroup$ @user2974951 Given the non-parametric distribution of the data and small sample size, I would like to make the comparison of the means as robust as possible. I guess a follow up question would be, do the t-test and F-test take into account uncertainty in the individual measurements? Or do they assume the measurements have no error? $\endgroup$
    – ken
    Commented Aug 21, 2019 at 11:48
  • $\begingroup$ What do you mean non-parametric distribution of the data? Also, parametric tests are more powerful than non-parametric tests such as bootstrapping. The t-test has no assumption on the distribution of the data. All measurements contain error, as long as they are not biased then it's ok. $\endgroup$ Commented Aug 21, 2019 at 11:52
  • $\begingroup$ @user2974951 sorry, I meant to say, non-normally distributed data. I generally associated parametric tests with assumptions on the distribution of the data (i.e., normally distributed), equal variances across the data sets. etc. I think I've found an monte carlo approach that addresses my issue. However, i will look into the t-test and F-test (and Mann-Whitney U test) you suggested. Thanks $\endgroup$
    – ken
    Commented Aug 21, 2019 at 12:05

1 Answer 1


My first attempt to analyze such data would be to use a two-sample Wilcoxon test.

Here are two nonnormal samples of size $n = 15$ to use for illustration.

x1 = round(rgamma(15, 6, 1/.5),2)
x2 = round(rgamma(15, 5, 1/.4),2)

 [1] 3.68 2.18 5.99 1.91 3.17 3.40 2.66 4.98 1.60
[10] 4.99 2.62 3.88 1.07 2.59 3.10
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.070   2.385   3.100   3.188   3.780   5.990 
 [1] 2.72 2.61 3.16 1.18 0.83 1.83 4.73 3.59 1.49
[10] 1.04 0.95 2.45 3.18 1.28 1.86
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.830   1.230   1.860   2.193   2.940   4.730 
boxplot(x1, x2, col="skyblue2", pch=19)

enter image description here

The two-sample Wilcoxon test finds that that 'locations' are not the same.

wilcox.test(x1, x2)

        Wilcoxon rank sum test

data:  x1 and x2
W = 163, p-value = 0.03672
alternative hypothesis: 
   true location shift is not equal to 0

You might argue that you want a straightforward test of the difference in sample means. Also, the two samples have somewhat different shapes, so perhaps the assumptions of the Wilcoxon test are not met.

To meet these objections we could use a permutation test with difference in means as the 'metric'. In many iterations we assign 15 of the observations to Group 1 and the rest to Group 2, and find the difference in means. This gives a simulation of the permutation distribution of the difference in means. The P-value of the (2-sided) permutation test is about 0.04 < 0.05.

d.obs = mean(x1) - mean(x2);  d.obs
[1] 0.9946667

x =c(x1, x2); g = as.factor(rep(1:2, each=15))

m = 10^5;  d.prm = numeric(m)
for(i in 1:m) {
  g.prm = sample(g)
  d.prm[i] = mean(x[g.prm==1]) - mean(x[g.prm==2]) }
mean(abs(d.prm) >= abs(d.obs))
[1] 0.03844

hist(d.prm, prob=T, col="skyblue2", 
     main="Simulated Permuatation Distribution")
pm = c(-1,1);  abline(v = pm*d.obs, col="red")

enter image description here

Similar results might be obtained, using an appropriate style of bootstrap. However, for a simple 2 sample test on samples as small as these, my personal preference is to use a permutation test.

Note1: (1) Gamma-distributed data were simulated as follows:

x1 = round(rgamma(15, 6, 1/.5), 2)  # pop mean is 3
x2 = round(rgamma(15, 5, 1/.4), 2)  # pop mean is 2

(2) For these particular data, a Welch t test gives about the same P-value as the Wilxoxon and permutation tests shown above:

t.test(x1, x2)$p.val
[1] 0.03834002

(3) here is one style of 95% nonparametric boostrap CI for the difference in population means: $(0.107, 1,859).$ It does not include $0.$

d.obs = mean(x1) - mean(x2)
d.re = replicate(10^4, 
   mean(sample(x1,15,rep=T)) - 
   mean(sample(x2,15,rep=T)) - d.obs) 
d.obs - quantile(d.re, c(.975,.025))
    97.5%      2.5% 
0.1073333 1.8593500 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.