Let's say I got two samples from two unknown random variables:

$(x_i), (y_i)$

At this stage I don't know (or don't want to assume) that they come from a similar process or not.

I want to compute a confidence interval of the expected difference $E[Delta] = E[X - Y]$.

Given the large size of the samples I suspect we can assume that the expected difference would be gaussian.

Therefore I imagine I can use the standard approach that builds the CI like this: $\mu +/- z^\alpha\sigma$ where $z^\alpha$ is the value of the inverse gaussian CDF at the given confidence level.

Where I'm pretty stuck is basically how to estimate the mean and standard deviation of the expected difference $E[X - Y]$.


Normally when estimating a confidence interval I think of an interval estimate for a population parameter (such as the mean). So when you say you want to compute a confidence interval for the difference, I assume you mean an interval estimate for the expected difference. Is this correct?

whether or not X-Y is Gaussian depends on the distribution of X and Y. If both X and Y are Gaussian, than X-Y will be Gaussian. if not, than X-Y will, in all likelihood, not be Gaussian.

When you're bootstrapping I am guessing that on each iteration you store the sample average difference. So you're really developing an interval for the estimate of $E(X-Y)$. Lets call this estimate $\bar{d}$.

In the parametric approach $\bar{d}$ will be the sample average of $X-Y$. $\bar{d}$ WILL be normally distributed by the central limit theorem, regardless of the distribution of $X$ and $Y$ assuming they are both iid. So even though X-Y is not Gaussian, $\bar{d}$ will be (at least in the limit as the number of observations approaches infinity).

Assuming $X$ and $Y$ are iid, $X-Y$ will have mean $E(X-Y)=E(X) - E(Y)$ which can be approximated by the sample average $\bar{d}$. The variance of this estimate can be derived in the following matter.

$$ var(\bar{d}) = var(\frac{1}{n}\sum_{i=1}^{n}(x_i - y_i))=...=\frac{\sigma^2}{n} $$ Where $\sigma^2=var(X-Y)=var(X)+var(Y)-2cov(X,Y)$. So roughly speaking $$ \bar{d} \sim n(E(X-Y) , \frac{\sigma^2}{n}) $$ and more formally $$ \sqrt(n)(\bar{d}-E(X-Y)) \rightarrow n(0,\frac{\sigma^2}{n})\;\;\;(in\;distribution) $$ Of course we don't know $\sigma^2$ so we instead approximate it with the sample variance $s^2$.

You can then form a confidence interval for $E(X-Y)$ with the above Gaussian distribution and it will hold asymptotically. I suppose you could also use the t-distribution in developing the confidence interval, I would not know off hand what the degrees of freedom would be (best guess n-1).

Hope that helps!

  • $\begingroup$ How does this go when X and Y are not iid ? or more precisely, in the context of an AB test for instance we don't know whether X and Y are iid so how to estimate the variance ? $\endgroup$ – oDDsKooL Dec 9 '14 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.