How to estimate the confidence interval of the difference of means?

I have two sets of measurements (repeated measurements of two processes, let's call them $$X=\{X_i\}_{0...n}$$ and $$Y=\{Y_i\}_{0...m}$$), and I would like to estimate (with confidence interval) the difference in means between the two underlying distributions.

The distributions are independently sampled. However, I would like to avoid making any other assumptions, such as equal variance, and normality if possible.

I first looked at this first course, but it assumes the sampling distributions are normally distributed, which may not hold in my case. I also looked at another course, which has also the same issue (assuming normal distribution). In addition, it seems the two courses disagree on the computation of the degrees of freedom.

I have been told to have a look at the jackknife method. However, I see how to use it to get an estimate of the mean of a sample of a single definition, but I am at a loss on how to apply the technique for comparing two distributions.

For example, if I am to use the jackknife method, should I construct the partial predictions by removing one observation from one sample each time? one observation from both samples?

• How many observations in each group? Can we assume independent sampling within each group? or correlation in time? Can you show us some plots? ... Mar 14, 2019 at 20:06
• @kjetilbhalvorsen yes, we can assume independent sampling within each group, because the process itself ensures independent samples. I cannot show you the plots unfortunately, but the samples are definitely not "normal looking". Mar 15, 2019 at 9:06
• As to observations within each group: it depends, but usually around 50. Mar 15, 2019 at 9:31
• So then I support the answer already given, you could try bootstraping Mar 15, 2019 at 9:44

Let $$\mu_X$$ and $$\mu_Y$$ denote the means of $$X$$ and $$Y$$ respectively. If the goal is to obtain a confidence interval for $$(\mu_X-\mu_Y)$$, Jackknife and bootstrap would both work well, assuming your observations are independently sampled, i.e. $$cov(X_i,X_j)=cov(Y_i,Y_j)=0$$ for all $$i \neq j$$. Let us denote the difference in means as $$D$$, so $$D=\mu_X-\mu_Y.$$

Bootstrap: The process consists of drawing $$B$$ bootstrap 'samples' for each variable, indexed by $$b=1,...,B$$. The steps are as follows:

1. Sample (with replacement) $$n$$ draws from your sample of observations of $$X$$ and $$m$$ draws from your sample of observations of $$Y$$, where $$n$$ and $$m$$ are the respective sample sizes in your data.

2. Calculate the sample means from your bootstrap 'sample' to estimate $$\mu_X$$ and $$\mu_Y$$ and take the difference in sample means to get an estimate for $$D$$. Denote this as $$\hat{D_1}$$.

3. Repeat steps 1 and 2 B times and save the estimated values of $$D$$, denoted $$\hat{D}_b$$ for bootstrap sample $$b$$, for $$b=1,...,B.$$ After completing this step, you have $$B$$ values of estimates for $$D$$, given by $$\{\hat{D}_1,\hat{D}_2,...,\hat{D}_B\}$$.

4. The $$(1-\alpha)$$% confidence interval is then given by $$(\hat{D}_{[\alpha B]},(\hat{D}_{[(1-\alpha) B]}$$), where $$\hat{D}_{[r]}$$ denotes the $$r$$th order statistic from our set of $$B$$ estimates of $$D$$. In other words, the confidence interval is formed by the $$\alpha$$th and $$(1-\alpha)$$th percentiles from our set of $$B$$ estimates of $$D$$.

Jackknife: The jackknife follows a similar process, except the jackknife 'samples' are formed by leaving out one observation at a time. The steps for the jackknife are as follows:

1. Drop one observation from your sample of $$X$$ and one observation from your sample of $$Y$$.

2. Calculate the sample means from your jackknife 'sample' to estimate $$\mu_X$$ and $$\mu_Y$$ and take the difference in sample means to get an estimate for $$D$$. Denote this as $$\hat{D_1}$$.

3. Repeat steps 1 and 2 for all pairs of observations of $$X$$ and $$Y$$. Since there are $$n$$ observations of $$X$$ and $$m$$ observations of $$Y$$, there are $$mn$$ unique pairs of observations from the two samples. For each jackknife sample $$j$$, save the estimated value of $$D$$, denoted $$\hat{D}_j$$ for bootstrap sample $$B$$, for $$j=1,...,mn$$. After completing this step, you have $$mn$$ values of estimates for $$D$$, given by $$\{\hat{D}_1,\hat{D}_2,...,\hat{D}_{mn}\}$$.

4. The $$(1-\alpha)$$% confidence interval is then given by $$(\hat{D}_{[\alpha B]},(\hat{D}_{[(1-\alpha) B]}$$), where $$\hat{D}_{[r]}$$ denotes the $$r$$th order statistic from our set of $$mn$$ estimates of $$D$$. In other words, the confidence interval is formed by the $$\alpha$$th and $$(1-\alpha)$$th percentiles from our set of $$mn$$ estimates of $$D$$.

• Both the jackknife and the bootstrap are relatively easy to conceptualize and implement. Good luck!
– dlnB
Mar 15, 2019 at 15:42