I want to bound the difference between two variables sampled from two different populations. When it's just one population with paired measurements a.k.a. "paired samples" (e.g. before treatment vs after treatment), I simply subtract the paired measurements and use univariate statistical inference. Is there a reason I cannot do the same for independent samples? Say I have two i.i.d. samples $X_1,X_2,...,X_n$ and $Y_1,Y_2,...,Y_m$ with respective cumulative distribution functions $F_X$ and $F_Y$. Of course, if I construct a set from all possible pairwise differences, the set is not i.i.d.; $X_1-Y_1$ and $X_1-Y_2$ are dependent, for example. However, if I construct a set where each random variable from the original two samples only appears in at most one difference, such as ${X_1-Y_1,X_2-Y_2,...,X_{min(n, m)}-Y_{min(n, m)}}$, is that set i.i.d. with cumulative distribution function $F_{X-Y}$? If so, can I then apply univariate statistics to that?
UPDATE: Based on the two methods of constructing a set from two independent samples, I simulated 10,000 confidence intervals for the
median bound by order statistics and 10,000 tolerance intervals bound
by order statistics for the difference between two normal random
variables. I stuck to normal variables because I could derive the median and cdf to test if a particular simulated interval really did contain the median or population proportion. The dependent nXm pairwise differences method yielded intervals that contained the median/proportion far less frequently than the intervals should have for iid samples, which is expected for a non-iid sample, and the min(n,m) differences method contained the median/proportion about as frequently as the interval should have for iid samples. I'm working on a way to do the same for non-normal populations next, but it's clear from this that the n*m non-iid sample wouldn't work for order statistics-based intervals.
3 Answers
In general if a family of $X_i$ are $iid$, then so so are $f(X_i)$. In your case, you're defining a new random variable $Z_i:=X_i-Y_i$, in which case $Z_i$ form an $iid$ sequence.
-
$\begingroup$ What I find weird is that if it's so simple to compute an iid sample of $X-Y$ from iid samples of $X$ and $Y$, then why isn't it used in studies that need to compare independent populations? I only see people comparing population means using Welch's t-test and constructing prediction or tolerance intervals for each population's sample to get an idea of the range. $\endgroup$ Commented Oct 4, 2016 at 21:58
-
$\begingroup$ Usually people tack on additional assumptions such as normality of X and Y, in which case if you add in independence of X, Y you immediately get X-Y is distributed $N(\mu_x-\mu_y,\sigma^2_x+\sigma^2_y)$. Add in sample estimates of both variances and that's how you get the Welch's t-statistic. However, and I think you're hinting at this, if we don't want to make distributional assumptions on X and Y, you can bootstrap X-Y by taking your observations and evaluating X-Y at each of the n x m possibilities, essentially empirically simulating the joint cdf $F_{X,Y}(X\leq x,Y \leq y)$ $\endgroup$– RayCommented Oct 4, 2016 at 22:50
-
$\begingroup$ Correct deduction, Ray! I am indeed planning to use nonparametric methods, specifically S.S. Wilk's tolerance interval bound by order statistics. It's also why I can't bootstrap X-Y by computing all nxm possibilities because there's a requirement for i.i.d., and as I've noted in the original post, the nxm set contains inter-dependent variables. $\endgroup$ Commented Oct 5, 2016 at 16:31
Briefly, to answer your first sentence: if X and Y are independent, then X-Y has expectation EX-EY, and its variances add only because of independence, so X-Y has variance var(X)+var(Y). You could bound the difference through e.g. Chebyshev's inequality, although these bounds are usually too wide to be of use.
What you are proposing is essentially a calculation of the empirical distribution of X-Y by going over all n by m combinations of X and Y, which also works and might be a bit stronger if you're looking to bound quantiles of the difference, for example. I'm less sure about the finite sample properties of bootstrap though.
Your first example taking all n x m combinations is the 'correct' way to proceed; I don't recommend the second method as you're essentially throwing out potential observations of the bootstrap. Then you can find the $j/(nm)$-th quartile by ordering your $nm$ observations and taking the j-th smallest value. In particular, the number of observations of each $X_i=k$ is directly proportional to the probability density $f_X(k)$, and similarly for $Y_j$, and by independence the probability of observing them both jointly is proportional to the product of the marginals. So you have to go through all n x m combinations to construct the empirical population of $X_i-Y_j$. Your second proposal has a total of m (or n) data points; both methods have the same properties but you're throwing out so much information by doing so. The only reason to do that is if, as you remarked, you have a natural pairing, in which case the difference of $X_i-Y_i$ allows you to cancel out an $i$-level effect, i.e. you have correlation between $X_i$ and $Y_i$.
Also, your use of 'iid' is a bit confused. iid only makes sense if you're talking about random variables; $X_1-Y_1$ and $X_1-Y_2$ aren't random; they're observed because you already have the data on the $X_i,Y_i$. If you have a program that picks a random $X_i$ and $Y_j$ and subtracts them, then that variable $X_i-Y_j$ where there are no restrictions on $i$ or $j$ is iid.
-
$\begingroup$ I thought random samples are sets of random variables. After all, the sample mean is considered a random variable, and it's just the sum of all members of the random sample divided by the constant sample size. If the random sample contained observed constants, then the sample mean has to be a constant. $\endgroup$ Commented Oct 4, 2016 at 17:41
-
$\begingroup$ The random sample is a random variable; however once it is observed it is no longer random. We can have degenerate random variables (like a 2-sided coin) where it's a random variable before you realize it and no longer a random variable afterwards, although there is no 'randomness'. Notationally $X_1-Y_1$ can be a r.v. (before running the experiment, I consider the properties of the differences of my first 2 obs), but in your case you're referring to observed values of $X_1$ and $Y_1$, which are not. $\endgroup$– RayCommented Oct 4, 2016 at 17:52
-
$\begingroup$ Then if the random sample is unobserved? I haven't really observed anything yet in real life, so that's still true. $\endgroup$ Commented Oct 4, 2016 at 17:56
-
$\begingroup$ I just edited my answer; hopefully this makes things a bit clearer. Random variables can be iid, the concept of iid does not extend to observations which are by definition not random (you already flipped a coin, the outcome cannot be random anymore) $\endgroup$– RayCommented Oct 4, 2016 at 18:35
-
$\begingroup$ I don't understand this answer at all. By assumption $X_i$ and $Y_i$ are random variables. Therefore $X_i-Y_i$ is also a random variable. $\endgroup$– Alex R.Commented Oct 4, 2016 at 18:52
Your defined set of differences will be i.i.d. Regarding the distribution of the differences, you can estimate it using the empirical cdf, non parametrically, if you don't know the underlying distributions. If X and Y are not exceedingly skewed, it will give you a good approximation.
